{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 梯度不平稳性与Glorot条件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从链式求导的角度来看，梯度消失和梯度爆炸的问题主要发生在使用反向传播算法进行训练的过程中。反向传播算法涉及到对损失函数相对于每个权重的梯度计算，而这些梯度是通过链式法则（Chain Rule）得到的。\n",
    "\n",
    "#### 链式法则\n",
    "\n",
    "在神经网络中，输出相对于输入的梯度可以分解为一系列中间变量相对于它们的前驱的梯度的乘积。具体来说，如果有一个由多层组成的神经网络，第$ l $层的输出依赖于第$ l-1 $层的输出，那么我们可以写成：\n",
    "\n",
    "$$ \\frac{\\partial L}{\\partial w_l} = \\frac{\\partial L}{\\partial z_l} \\cdot \\frac{\\partial z_l}{\\partial w_l} $$\n",
    "\n",
    "其中：\n",
    "- $ L $ 是损失函数。\n",
    "- $ w_l $ 是第$ l $层的权重。\n",
    "- $ z_l $ 是第$ l $层的线性组合输出（通常是$ w_l \\cdot a_{l-1} + b_l $，其中$ a_{l-1} $是第$ l-1 $层的激活）。\n",
    "\n",
    "对于更深层次的情况，我们有：\n",
    "\n",
    "$$ \\frac{\\partial L}{\\partial w_{l-1}} = \\frac{\\partial L}{\\partial z_l} \\cdot \\frac{\\partial z_l}{\\partial a_{l-1}} \\cdot \\frac{\\partial a_{l-1}}{\\partial z_{l-1}} \\cdot \\frac{\\partial z_{l-1}}{\\partial w_{l-1}} $$\n",
    "\n",
    "以此类推，直到输入层。这里，$\\frac{\\partial z_l}{\\partial a_{l-1}}$ 和 $\\frac{\\partial a_{l-1}}{\\partial z_{l-1}}$ 分别对应了第$ l $层的权重矩阵和激活函数的导数。\n",
    "\n",
    "#### 成因分析\n",
    "\n",
    "**梯度消失：**\n",
    "\n",
    "当网络很深时，上述梯度的乘积可能包含许多项，每项都是一个激活函数的导数。如果使用的激活函数（如Sigmoid或Tanh）在输入远离0的情况下导数接近0，则乘积中的很多因子都会很小。随着深度增加，这种效应会累积，最终导致梯度变得非常小，这就是梯度消失。\n",
    "\n",
    "**梯度爆炸：**\n",
    "\n",
    "与此相反，如果乘积中的某些因子变得非常大（例如，当激活函数的导数在某些区域很大时），或者权重本身非常大，那么整个乘积也可能变得非常大，导致梯度爆炸。\n",
    "\n",
    "总结一下，不同层参数的梯度在计算过程中都有很大的差异，并且这种差异是一种累乘效应，我们也可以简单理解为是一种伴随着层数增加指数级变化的差异。而这种累乘效应会导致线性层参数的一部分梯度过大而另一部分过小，从而影响模型平稳训练。而从具体原因来说，每一层参数的梯度主要和两个因素相关，其一是线性层输入数据，其二则是激活函数导函数计算结果。\n",
    "\n",
    "#### 影响因素\n",
    "\n",
    "梯度消失和梯度爆炸的影响因素包括但不限于以下几点：\n",
    "\n",
    "- **激活函数的选择**：不同的激活函数有不同的导数特性。例如，Sigmoid和Tanh函数在输入远离原点时导数趋近于0，而ReLU函数在正半轴上的导数恒为1，这有助于缓解梯度消失。\n",
    "- **网络的深度**：更深的网络意味着更多的梯度乘积项，更容易遇到梯度消失或爆炸的问题。\n",
    "- **权重初始化**：不恰当的权重初始化可能导致梯度问题，比如太大的权重可能引发梯度爆炸，而太小的权重可能加剧梯度消失。\n",
    "- **正则化技术**：如L1或L2正则化可以帮助控制权重大小，从而间接地影响梯度的大小。\n",
    "- **梯度裁剪**：可以手动设定一个阈值，将梯度裁剪到一个合理范围内，防止梯度爆炸。\n",
    "\n",
    "通过调整这些因素，可以有效地缓解梯度消失和梯度爆炸带来的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、Sigmoid和tanh激活函数的梯度更新问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.Sigmoid激活函数的梯度消失问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 理论说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;对于sigmoid激活函数来说，简单的叠加是极容易出现梯度消失的问题。sigmoid函数及导函数图像如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 绘制Sigmoid函数的函数图像和导函数图像\n",
    "X = torch.arange(-5, 5, 0.1)\n",
    "X.requires_grad=True\n",
    "sigmoid_y = torch.sigmoid(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 反向传播\n",
    "sigmoid_y.sum().backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Sigmoid Derivative function')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAAGxCAYAAAC5hxYeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABzU0lEQVR4nO3deXhTVfoH8G+SNkn30r2FrlBadkrZCiKbgoDAqAwoyiKgMriw6G8GxlEUZ0ScUXEDl0EYZxRQARFFEZWdskoRKXspLd1b6N6mbXJ+f6Q3ENpCU9LeJP1+nifPAzf3Ju9Nmps357znHIUQQoCIiIjIRijlDoCIiIjoekxOiIiIyKYwOSEiIiKbwuSEiIiIbAqTEyIiIrIpTE6IiIjIpjA5ISIiIpvC5ISIiIhsCpMTIiIisilMTprBwYMHcd999yEsLAwajQaBgYFISEjAs88+a7bfkCFDMGTIEHmCbKTU1FQoFAqsWbPmlvu+9NJLUCgUt9xv+vTpUCgU9d6+/fZbK0TddK+++iq+/vrrOtt37twJhUKBnTt3tnhM1DT8HN7cjZ9DNzc3REREYNy4cVi9ejV0Op0VIq//eSMiIprlsSUrVqyo97Wy5HW0ttTUVIwZMwY+Pj5QKBSYN29ei8cgKS8vx0svvVTv9WzNmjVQKBRITU1t8biu5yTrszug7777DuPGjcOQIUPw+uuvIzg4GFlZWThy5AjWrVuHN954w7TvihUrZIy0cYKDg5GYmIj27dtb9XFdXFzwyy+/1NkeGxtr1eex1KuvvooJEybgD3/4g9n2Xr16ITExEZ07d5YnMLIIP4eNc/3nsKKiAunp6fj+++/x2GOP4Y033sAPP/yAdu3aWfU5X3jhBcydO9eqj3mjFStWwM/PD9OnTzfb3lyvY2PMnz8fBw8exCeffIKgoCAEBwe3eAyS8vJyvPzyywBQJzEfM2YMEhMTZY0PACDIqu68807Rvn17UV1dXec+vV4vQ0QtZ/HixaIxf1LTpk0Tbm5uLRCR5dzc3MS0adPkDoNuEz+Ht/c53LZtm3B2dhb9+vWzWlxlZWVWe6xb6dKlixg8eHCLPV9jdOjQQYwaNUruMIQQQuTl5QkAYvHixXKH0iB261hZQUEB/Pz84ORUt1FKqTR/uetrTr58+TImTJgADw8PeHt74+GHH8bhw4frNEVOnz4d7u7uOH36NEaOHAk3NzcEBwfjtddeAwAcOHAAd9xxB9zc3NCxY0f85z//qRPP77//jvHjx6NNmzbQarXo2bNnnf0aagb97rvv0LNnT2g0GkRGRuJf//qXBa/SzTXUhVJfLNLrcP78eYwePRru7u4IDQ3Fs88+W6dZWqfTYcmSJejUqRO0Wi18fX0xdOhQ7N+/HwCgUChQVlaG//znP6ambun9aSimb775BgkJCXB1dYWHhwfuvvtuJCYmmu0jNbOfPHkSDz30ELy8vBAYGIgZM2agqKjIKq8ZmePn8PaMGDECjz32GA4ePIjdu3eb3bd+/XokJCTAzc0N7u7uGDlyJI4dO2a2j/S6nDhxAiNGjICHhweGDx9uuu/6bp24uDgMGjSoTgx6vR5t27bF/fffb9r28ssvo1+/fvDx8YGnpyd69eqFVatWQVy3fm1ERAROnjyJXbt2mT7H0vPd+Dp+/fXXUCgU+Pnnn+s8/8qVK6FQKPDbb7+Zth05cgTjxo2Dj48PtFot4uLi8MUXX9z0tZSuHefPn8f3339viik1NbXBLpT6rjdDhgxB165dcfjwYQwaNAiurq6IiorCa6+9BoPBYHZ8YWEhnn32WURFRUGj0SAgIACjR4/G6dOnkZqaCn9/f9PrKcUjtTI1FNMnn3yCHj16QKvVwsfHB/fddx9OnTplto8l1+NbYXJiZQkJCTh48CCeeeYZHDx4ENXV1Y0+tqysDEOHDsWOHTuwbNkyfPHFFwgMDMSkSZPq3b+6uhr3338/xowZg82bN2PUqFFYtGgR/vrXv2LatGmYMWMGNm3ahJiYGEyfPh1Hjx41HXvmzBkMGDAAJ0+exDvvvIONGzeic+fOmD59Ol5//fWbxvnzzz9j/Pjx8PDwwLp16/DPf/4TX3zxBVavXt3ocwWAmpoas5ter7fo+Otfh3HjxmH48OHYvHkzZsyYgbfeegvLli0ze65Ro0bhlVdewb333otNmzZhzZo1GDBgANLS0gAAiYmJcHFxwejRo5GYmIjExMSbNvl//vnnGD9+PDw9PbF27VqsWrUKV69exZAhQ7B37946+z/wwAPo2LEjNmzYgIULF+Lzzz/H/Pnzm3TOdHP8HN6+cePGAYBZcvLqq6/ioYceQufOnfHFF1/gv//9L0pKSjBo0CAkJyebHV9VVYVx48Zh2LBh2Lx5s6kb4UaPPvoo9u7di3Pnzplt//HHH5GZmYlHH33UtC01NRVPPPEEvvjiC2zcuBH3338/nn76abzyyiumfTZt2oSoqCjExcWZPsebNm2q97nvvfdeBAQE1PuarVmzBr169UL37t0BADt27MDAgQNRWFiIDz74AJs3b0bPnj0xadKkm9awSF3CQUFBGDhwoCmmpnSbZGdn4+GHH8YjjzyCb775xvS39r///c+0T0lJCe644w58+OGHePTRR7FlyxZ88MEH6NixI7KyshAcHIwffvgBADBz5kxTPC+88EKDz7t06VLMnDkTXbp0wcaNG/H222/jt99+Q0JCQp33rTHX40aRu+nG0eTn54s77rhDABAAhLOzsxgwYIBYunSpKCkpMdt38ODBZk2P77//vgAgvv/+e7P9nnjiCQFArF692rRt2rRpAoDYsGGDaVt1dbXw9/cXAMSvv/5q2l5QUCBUKpVYsGCBaduDDz4oNBqNSEtLM3uuUaNGCVdXV1FYWCiEEOLixYt1nrtfv34iJCREVFRUmLYVFxcLHx+fRjcnS6/P9beBAwcKIYTYsWOHACB27Nhhdlx9sUiP9cUXX5jtO3r0aBETE2P6/6effioAiI8//vimsTXUrXNjTHq9XoSEhIhu3bqZdROUlJSIgIAAMWDAANM2qZn99ddfN3vMOXPmCK1WKwwGw01jIsvxc3j73aunTp0SAMSf/vQnIYQQaWlpwsnJSTz99NNm+5WUlIigoCAxceLEOq/LJ598Uu/zhoeHm/6fn58v1Gq1+Otf/2q238SJE0VgYGC9XXNCGD+D1dXVYsmSJcLX19fsc9RQt059r+OCBQuEi4uL6bUWQojk5GQBQLz77rumbbGxsSIuLq5OPPfee68IDg6+ZXdheHi4GDNmjNm21atXCwDi4sWLZtvruwYOHjxYABAHDx4027dz585i5MiRpv8vWbJEABDbt29vMJabdevcGNPVq1eFi4uLGD16tNl+aWlpQqPRiMmTJ5u2NfZ63BhsObEyX19f7NmzB4cPH8Zrr72G8ePH4+zZs1i0aBG6deuG/Pz8Bo/dtWsXPDw8cM8995htf+ihh+rdX6FQYPTo0ab/Ozk5oUOHDggODkZcXJxpu4+PDwICAnDp0iXTtl9++QXDhw9HaGio2WNOnz4d5eXldbomJGVlZTh8+DDuv/9+aLVa03YPDw+MHTu2wXO7kYuLCw4fPmx2W7VqVaOPv55Coajz3N27dzc73++//x5arRYzZsxo0nPc6MyZM8jMzMSUKVPMugnc3d3xwAMP4MCBAygvLzc7Rvolen2MlZWVyM3NtUpMdA0/h7dPXNdVAgDbtm1DTU0Npk6datbiqdVqMXjw4HpHfjzwwAO3fB5fX1+MHTsW//nPf0zdE1evXsXmzZsxdepUs665X375BXfddRe8vLygUqng7OyMF198EQUFBU3+HM2YMQMVFRVYv369advq1auh0WgwefJkAMD58+dx+vRpPPzwwwDMW31Hjx6NrKwsnDlzpknPb4mgoCD07dvXbFt917qOHTvirrvusspzJiYmoqKiok5xcWhoKIYNG1anS6wx1+PGYHLSTHr37o2//OUv+PLLL5GZmYn58+cjNTX1pk21BQUFCAwMrLO9vm0A4OrqanZhAgC1Wg0fH586+6rValRWVpo9V33NiiEhIab763P16lUYDAYEBQXVua++bQ1RKpXo3bu32S0mJqbRx1+vvtdBo9GYnW9eXh5CQkLq1Bs0lfT6NPQaGgwGXL161Wy7r69vnRgB4ygJah78HDad9GUixZKTkwMA6NOnD5ydnc1u69evr5Pwubq6wtPTs1HPNWPGDGRkZGD79u0AgLVr10Kn05l9IR46dAgjRowAAHz88cfYt28fDh8+jOeffx5A0z9HXbp0QZ8+fUxdO3q9Hv/73/8wfvx403sonftzzz1X59znzJkDADdNeK3lxmsIYLyOXH/ueXl5Vh1hdatr3Y1/o425HjcGhxK3AGdnZyxevBhvvfUWfv/99wb38/X1xaFDh+psz87OtnpMvr6+yMrKqrM9MzMTAODn51fvcW3atIFCoag3JmvFKf1h31hAdTsffn9/f+zduxcGg8EqCYp0kWjoNVQqlWjTps1tPw9ZDz+Hlvnmm28AXBtqKsXy1VdfITw8/JbHN2auFcnIkSMREhKC1atXY+TIkVi9ejX69etnNnR/3bp1cHZ2xrfffmv25VffvESWevTRRzFnzhycOnUKKSkpyMrKMqt1kc590aJFZgW612vKj6vmutZdvny5ycff6FbXuob+Rm8XW06srL43EICpqln6FVKfwYMHo6SkBN9//73Z9nXr1lkvwFrDhw/HL7/8YroISj799FO4urqif//+9R7n5uaGvn37YuPGjWaZcElJCbZs2WKV2KTK+uur5IFrF8umGDVqFCorK285+dKNv0IaEhMTg7Zt2+Lzzz83a/4uKyvDhg0bTCN4SB78HN6e7du349///jcGDBiAO+64A4AxgXBycsKFCxfqtHpKt6ZSqVSYMmUKvv76a+zZswdHjhyp0wWrUCjg5OQElUpl2lZRUYH//ve/dR6vsZ9jyUMPPQStVos1a9ZgzZo1aNu2ramVBjB+3qOjo3H8+PEGz93Dw8Pi826ua93Zs2frnUdKYkmrbUJCAlxcXMyKbgHjiDapW7I5sOXEykaOHIl27dph7NixiI2NhcFgQFJSEt544w24u7vfdPKhadOm4a233sIjjzyCv//97+jQoQO+//57bNu2DUDdIZC3Y/Hixfj2228xdOhQvPjii/Dx8cFnn32G7777Dq+//jq8vLwaPPaVV17BPffcg7vvvhvPPvss9Ho9li1bBjc3N1y5cuW2YwsKCsJdd92FpUuXok2bNggPD8fPP/+MjRs3NvkxH3roIaxevRqzZ8/GmTNnMHToUBgMBhw8eBCdOnXCgw8+CADo1q0bdu7ciS1btiA4OBgeHh71/iJSKpV4/fXX8fDDD+Pee+/FE088AZ1Oh3/+858oLCw0DSUlefBz2LjPocFgwIEDBwAYf72npaXh+++/xxdffIFOnTqZDZONiIjAkiVL8PzzzyMlJQX33HMP2rRpg5ycHBw6dAhubm4NjshpjBkzZmDZsmWYPHkyXFxc6oyOGjNmDN58801MnjwZjz/+OAoKCvCvf/3L9EV7vW7dumHdunVYv349oqKioNVq0a1btwaf29vbG/fddx/WrFmDwsJCPPfcc3Xe5w8//BCjRo3CyJEjMX36dLRt2xZXrlzBqVOn8Ouvv+LLL7+0+Jz79OmDmJgYPPfcc6ipqUGbNm2wadOmekf7Nda8efOwfv16jB8/HgsXLkTfvn1RUVGBXbt24d5778XQoUPh4eGB8PBwbN68GcOHD4ePjw/8/PzqnbnX29sbL7zwAv76179i6tSpeOihh1BQUICXX34ZWq0WixcvbnKsN2VR+Szd0vr168XkyZNFdHS0cHd3F87OziIsLExMmTJFJCcnm+174ygBIYwV0Pfff79wd3cXHh4e4oEHHhBbt24VAMTmzZtN+zVUaT948GDRpUuXOtvrqxQ/ceKEGDt2rPDy8hJqtVr06NHDrIpdiPqr24UQ4ptvvhHdu3cXarVahIWFiddee82qk7BlZWWJCRMmCB8fH+Hl5SUeeeQRceTIkXpHS9T3WPXFUlFRIV588UURHR0t1Gq18PX1FcOGDRP79+837ZOUlCQGDhwoXF1dBQDT+9PQCKKvv/5a9OvXT2i1WuHm5iaGDx8u9u3bV28seXl5ZtsbqtSn28fPoeWj5lxcXERYWJgYO3as+OSTT4ROp6v3uK+//loMHTpUeHp6Co1GI8LDw8WECRPETz/9dMvXRbrv+tE61xswYIAAIB5++OF67//kk09ETEyM0Gg0IioqSixdulSsWrWqzucoNTVVjBgxQnh4eAgApudr6HUUQogff/zR9FqcPXu23uc/fvy4mDhxoggICBDOzs4iKChIDBs2THzwwQf17n+9+t57IYQ4e/asGDFihPD09BT+/v7i6aefFt999129o3Xq+5uq7/W8evWqmDt3rggLCxPOzs4iICBAjBkzRpw+fdq0z08//STi4uKERqMRAEyjFBu6Lv373/82/a15eXmJ8ePHi5MnT9aJpbHX41tRCHFDSTbZnFdffRV/+9vfkJaWZvWppImocfg5JGo57NaxMe+99x4A4xoz1dXV+OWXX/DOO+/gkUce4QWRqIXwc0gkLyYnNsbV1RVvvfUWUlNTodPpEBYWhr/85S/429/+JndoRK0GP4dE8mK3DhEREdkUDiUmIiIim8LkhIiIiGwKkxMiIiKyKXZREGswGJCZmQkPDw+LpkQmIusQQqCkpMSq6xM1N143iOTX1GuHXSQnmZmZdVbtJKKWl56ebjdDaXndILIdll477CI5kdYsSE9Pb/Qql0RkPcXFxQgNDW3S+iFy4XWDSH5NvXbYRXIiNcl6enryIkMkI3vqHuF1g8h2WHrtsI/OYyIiImo1mJwQERGRTWFyQkRERDaFyQkRERHZFCYnREREZFOYnBAREZFNYXJCRERENsXi5GT37t0YO3YsQkJCoFAo8PXXX9/ymF27diE+Ph5arRZRUVH44IMPmhIrERERtQIWJydlZWXo0aMH3nvvvUbtf/HiRYwePRqDBg3CsWPH8Ne//hXPPPMMNmzYYHGwRERE5PgsniF21KhRGDVqVKP3/+CDDxAWFobly5cDADp16oQjR47gX//6Fx544AFLn56IiIgcXLPXnCQmJmLEiBFm20aOHIkjR46gurq63mN0Oh2Ki4vNbkRk21asWIHIyEhotVrEx8djz549De67ceNG3H333fD394enpycSEhKwbds2s33WrFkDhUJR51ZZWdncp0JEMmv25CQ7OxuBgYFm2wIDA1FTU4P8/Px6j1m6dCm8vLxMN64sSmTb1q9fj3nz5uH555/HsWPHMGjQIIwaNQppaWn17r97927cfffd2Lp1K44ePYqhQ4di7NixOHbsmNl+np6eyMrKMrtptdqWOCUiklGLLPx344I/Qoh6t0sWLVqEBQsWmP4vrWpIRLfHYBDILdEhtaAMaVfKcflqBTKuVuCfE7pDqWz6on5vvvkmZs6ciVmzZgEAli9fjm3btmHlypVYunRpnf2lbl7Jq6++is2bN2PLli2Ii4szbVcoFAgKCmpUDDqdDjqdzvR/trjav4oqPbaeyMLV8iqM6R6MYC8XuUOiFtLsyUlQUBCys7PNtuXm5sLJyQm+vr71HqPRaKDRaJo7NCKHJYRARmEFkjOLcTanBGdySnEhtxQX88tQUa2vs///jYxBkFfTWiSqqqpw9OhRLFy40Gz7iBEjsH///kY9hsFgQElJCXx8fMy2l5aWIjw8HHq9Hj179sQrr7xilrxcb+nSpXj55ZebdA5kez7afQHv77iAogpj9//S709jdLdgvHpfV3honWWOjppbsycnCQkJ2LJli9m2H3/8Eb1794azM//AiKyhsLwKv6Zdxa+XCnH8ciF+u1xkuqjfSKVUoF0bF4T5uKJdG1eE+rhA7dT0Ht78/Hzo9fp6u29v/GHSkDfeeANlZWWYOHGiaVtsbCzWrFmDbt26obi4GG+//TYGDhyI48ePIzo6us5jsMXVcaw7lIZXt54GALRr44JgLy0Op17FluOZqKzW48NH4m+rpY9sn8XJSWlpKc6fP2/6/8WLF5GUlAQfHx+EhYVh0aJFyMjIwKeffgoAmD17Nt577z0sWLAAjz32GBITE7Fq1SqsXbvWemdB1MqU6Wpw8GIB9pzLR+KFApzOLqmzj5NSgehAD3QK8kB0oAc6BLijvb8bQn1c4ayyfrlZfd23DXXdXm/t2rV46aWXsHnzZgQEBJi29+/fH/379zf9f+DAgejVqxfeffddvPPOO3Uehy2ujuHopat4cfNJAMDTwzpg3l0doVIqcDClAFM+OYTtyTl455dzmHdXR5kjpeZkcXJy5MgRDB061PR/6ZfKtGnTsGbNGmRlZZkVwUVGRmLr1q2YP38+3n//fYSEhOCdd97hMGIiC2UXVeLH5GxsT87BwZQrqNIbzO6P8nNDr/A26BnqjR7tvNExyB0aJ1Wzx+Xn5weVSlVv9+2NrSk3Wr9+PWbOnIkvv/wSd9111033VSqV6NOnD86dO3fbMZNtqqjS48nPfkWV3oB7ugRh/l0dTS0k/aJ88Y8/dMX/ffUblv90DglRvugXVX9pANk/i5OTIUOGmApa67NmzZo62wYPHoxff/3V0qciavUKy6uw5bcsfJOUgcOpV83ua9fGBYOi/TGwgy/6RfrC30OeVgO1Wo34+Hhs374d9913n2n79u3bMX78+AaPW7t2LWbMmIG1a9dizJgxt3weIQSSkpLQrVs3q8RNtuezg5eQXVyJdm1c8MbEHnW6bv7YOxSHU6/giyOX8dZPZ7Hu8QSZIqXm1iKjdYio8YQQOJByBZ8dvIQfT+aYtZDEh7fBiM6BGN4pEO393RrVbdISFixYgClTpqB3795ISEjARx99hLS0NMyePRsA6nT3rl27FlOnTsXbb7+N/v37m1pdXFxc4OXlBQB4+eWX0b9/f0RHR6O4uBjvvPMOkpKS8P7778tzktSsKqv1+HB3CgDgqaEd4Kap/+tp3l0dselYBg6kXMGhi1fQN9Kn3v3IvjE5IbIRldV6bE7KwKq9F3E2p9S0vVOwJ+6Pa4t7e9juUMpJkyahoKAAS5YsQVZWFrp27YqtW7ciPDwcAOp093744YeoqanBk08+iSeffNK0XeoeBoDCwkI8/vjjyM7OhpeXF+Li4rB792707du3Rc+NWsb6w+nIK9GhrbcL7u/VrsH9Qrxd8Mfeofj8YBre/eUc/juzXwtGSS1FIW7WR2MjiouL4eXlhaKiInh6esodDpFVlVfV4PODafhwdwrySozzdLiqVfhDXFtM7huGrm29ZI7QPj+D9hhza1WtN+DO13cgq6gSr4zvgikJETfdP/1KOYb+aydqDAKb5gxAXFiblgmULNbUzyFbTohkUq03YN3hdLz90znklxqTkhAvLR4dGIlJfUPhybkcqJXYfTYPWUWV8HNX44+9bz38O9THFeN6hmDjrxlYfzidyYkDYnJCJIMdZ3LxypZkpOSXAQBCfVzw1NAOuC+u3W3NOUJkj75OygQAjO0RAq1z40aYTYhvh42/ZuC7E1l4aVyXRh9H9oHJCVELyiyswIubT+KnUzkAAD93NeYOj8akPmFMSqhVKtXVYHuysSD6vri2jT6uf6Qvgr20yCqqxM4zubina3BzhUgy4NWQqAUIIfC/A5cw4q3d+OlUDpyUCjx+ZxR2PDcEUxIimJhQq7Xt92xUVhsQ5eeGbhbUVymVCozrEQIA+PpYZnOFRzJhywlRM8stqcSfv/oNO8/kAQB6hXlj2QPdER3oIXNkRPL7OikDAPCHuLYWD43/Q1xbfLg7Bb+czkVReTW8XFmn5Sj4c42oGe09l49Ry/dg55k8aJyUePHezvhy9gAmJkQA8kp02Hc+HwAwvmeIxcd3CvZETKAHqvQG/HAyy9rhkYyYnBA1A4NB4P0d5zH1k4MoKKtCbJAHtjx9B2bcEQkVFywjAgDsPJMLgwC6tfVCuK9bkx7j3u7GWpOfT+VaMzSSGZMTIiurrNbjmXXH8M9tZ2AQwKTeofj6yYHoyNYSIjM7zxq7OofG+Df5MYbEGBeL3Hc+H1U1hlvsTfaCNSdEVlRQqsOsT4/gWFohnFUKvDK+Kx7sGyZ3WEQ2p0ZvwJ7a5GRwTMAt9m5YlxBP+LmrkV9ahSOXrmBAez9rhUgyYssJkZVkFFbgjx8k4lhaIbxcnPHpjH5MTIgakJReiOLKGni7OqNnqHeTH0epVODOjsaWl121Redk/5icEFlBSl4pJqzcj5T8MrT1dsHGOQOQ0J7LuRM1RBq9Nija/7brsKSunZ1MThwGkxOi23QxvwwPfnQAWUWVaO/vhi9nJ6C9v7vcYRHZtJ1njQWsQzo2vd5Ecme0H5QK4ExOCTILK2778Uh+TE6IbkNaQTke+ugAckt0iAn0wBdPJCDE2zZXDiayFbkllfg9oxgATF0yt8PbVW3qGtp1lq0njoDJCVET5ZZU4pFVB5FdXInoAHd89lg/+Lpr5A6LyOYlXigAYCxm9fewzmdGSnKkeVPIvjE5IWqCkspqPLr6MNKulCPMxxWfzeoHPyYmRI1yOPUKAKBfpPXqsqTHOpx6BUIIqz0uyYPJCZGFavQGPPn5MZzMLIafuxqfzuiLAE+t3GER2Y1DF43JSd9IH6s9ZlyYN5xVCuQU65B2pdxqj0vyYHJCZKF/bD2F3Wfz4OKswurpfRHh17SZLYlao6tlVTibUwoA6BPRxmqPq3VWoUc7bwDAwdrkh+wXkxMiC6w7lIbV+1IBAG9O7IFu7Rq/iioRXevSiQ5wt3qNltQSc5jJid1jckLUSCcuF+HFzScBAPPv6ohR3YJljojI/khdOn2s2KUjkR7zUCqTE3vH5ISoEYrKqzHn86Oo0htwd+dAPDO8g9whEdmlQ6ZiWOsnJ/HhbaBUAJcKypFdVGn1x6eWw+SE6BaEEPi/r44j/UoFQn1c8K8/9oBCwZWFiSxVqqvB7xlFAIA+EdZPTjy1zugc4gmArSf2jskJ0S2sP5yOH5Nz4KxSYMXkeHi5OMsdEpFd+vXSVRgE0K6NS7NNVtg3onZIMetO7BqTE6KbuJhfhpe3JAMA/m9kDAtgiW5DUnohAGP3S3ORHlt6LrJPTE6IGqA3CMxfn4SKaj0Sonwx644ouUMismvHaxOG21mF+FZ6hBp/QJzKKkZltb7ZnoeaF5MTogas3ncRSemF8NA44Y2JPaC8zZVTiVozIYSpNaNHMyYnbb1d4OeuRo1BIDmruNmeh5oXkxOielwqKMO/fjwDAPjrmE5czI/oNl2+WoGCsio4qxToHOzZbM+jUChMk7EdZ9eO3WJyQnQDIQQWbTyBymoDEqJ88WCfULlDIrJ7xy8XAgA6BXtC66xq1ueSWmZYd2K/mJwQ3eCb45nYf6EAWmclXnugG4cNE1lBUlohAJhaNZqTlJyw5cR+MTkhuk5JZTX+8d0pAMBTQzsg3Jfr5hBZg9Ry0pz1JpIetaPqUgvKUVhe1ezPR9bH5IToOm//dA65JTpE+LrisTs5OofIGmr0BpyonXytOUfqSLxd1YisXZDz+OWiZn8+sj4mJ0S1LuSVYvX+VADAS+O6QOPUvP3iRK3FmZwSVFYb4KFxQlQLreIttZ5I3UlkX5icENVauvU09AaB4bEBGBITIHc4RA7jt9rWi27tvFpsSL6p7qS2O4nsC5MTIgCJFwrw06kcqJQKLBrdSe5wiByKtJ5Ot7YtN8Ny19rnOpnJbh17xOSEWj2DQeDVrcYi2Ml9w9AhwF3miIgcy8lM42RoXVowOekU7AmFAsgp1iG/VNdiz0vWweSEWr3vf8/GiYwiuGucMPeuaLnDIXIoeoPA6eza5CSk+SZfu5G7xgmRtaPtpOSI7AeTE2rV9AaBN7cbZ4KdNSgSfu4amSMiciwpeaWorDbAVa0yJQstpXNtMsSuHfvD5IRatU3HMnAhrwzers6YeUek3OEQORyp1aJTsGeLr0/VJUSqO2HLib1hckKtVlWNAW//fBYAMHtwe3honWWOiMjxSK0WLdmlI5Ge82QGW07sDZMTarW+PpaB9CsV8HPXYGpCuNzhEDkkqdWia0jLFcNKpOQktaAcJZXVLf781HRMTqhV0hsEVu66AAB4/M5IuKqdZI6IyPEIIUzJSWcZWk583TUI9tICAE5llbT481PTMTmhVum7E1m4mG+sNXm4H1tNiJpDRmEFiiqq4axSoGOghywxdGFRrF1ickKtjsEg8P4v5wEAjw6IhJuGrSZEzeH3DGOrSXSAB9RO8nzddK7tTpJiIfvA5IRanR1ncnEmpwTuGidMHxAhdzhEDutUlnxdOhKp5SQ5i8mJPWFyQq3Ox3tSAACT+4XBy5UjdIiaizT5Wqdg+ZKTTkHG576QW4pqvUG2OMgyTE6oVfk9owgHUq7ASalgqwlRMzudbSxC7RQkT70JALRr4wI3tQpVegNS88tki4Msw+SEWhWp1WRM92CEeLvIHA2R4yrT1SDtSjkAIEbG5ESpVJie/1Q2R+zYCyYn1GpkFlbg29+yAACPDYqSORoix3Y2pwRCAP4eGvjKvCxETG3XzmnWndgNJifUavzvwCXoDQL9In1My6kTUfOQunRiZWw1kXQKNsZwhi0ndoPJCbUKldV6rDucDgB4dGCEvMEQtQJnbCg5iZVaTpic2A0mJ9QqbD2RhStlVQj20uKuToFyh0Pk8KRhxFJiIKeY2gngMgorUMxp7O0CkxNqFf6TeAkA8HC/MDip+GdP1JyEEKZWCjmLYSVers4IqZ3Gnl079oFXaXJ4v10uxPH0QqhVSjzYN0zucIgcXk6xDkUV1VApFegQ4C53OACA2GB27dgTJifk8D4/mAYAGN0tCH4yjxogag1O1U6+FunnBq2zSuZojKQWHI7YsQ9MTsihlepq8M3xTADAQ2w1IWoRp7NspxhWIsXClhP70KTkZMWKFYiMjIRWq0V8fDz27Nlz0/0/++wz9OjRA66urggODsajjz6KgoKCJgVMZIktxzNRXqVHlL8b+kb6yB0OUatwNqe23kSmlYjrI7WcnM0ugRBC5mjoVixOTtavX4958+bh+eefx7FjxzBo0CCMGjUKaWlp9e6/d+9eTJ06FTNnzsTJkyfx5Zdf4vDhw5g1a9ZtB090K+sOGf8uH+wTCoVCIXM0RK2DlJx0tKGWkyg/dzgpFSjR1SC7uFLucOgWLE5O3nzzTcycOROzZs1Cp06dsHz5coSGhmLlypX17n/gwAFERETgmWeeQWRkJO644w488cQTOHLkyG0HT3QzyZnFOH65CM4qBR7o1U7ucIhaBb1B4HxuKQCgow21nKidlIjwcwMAnM0plTkauhWLkpOqqiocPXoUI0aMMNs+YsQI7N+/v95jBgwYgMuXL2Pr1q0QQiAnJwdfffUVxowZ0+Dz6HQ6FBcXm92ILPXFEeOkayM6B8k+fTZRa5F+pRy6GgM0TkqE+bjKHY6ZjoHGkUPnclh3YussSk7y8/Oh1+sRGGg+iVVgYCCys7PrPWbAgAH47LPPMGnSJKjVagQFBcHb2xvvvvtug8+zdOlSeHl5mW6hoaGWhEmEqhoDNidlAAAm9GarSUuwpBZt48aNuPvuu+Hv7w9PT08kJCRg27ZtdfbbsGEDOnfuDI1Gg86dO2PTpk3NeQpkBVKXTnt/d6iUttWVGh3AaeztRZMKYm/suxdCNNifn5ycjGeeeQYvvvgijh49ih9++AEXL17E7NmzG3z8RYsWoaioyHRLT09vSpjUiu04k4ur5dXw99BgUAc/ucNxeJbWou3evRt33303tm7diqNHj2Lo0KEYO3Ysjh07ZtonMTERkyZNwpQpU3D8+HFMmTIFEydOxMGDB1vqtKgJztV26djC5Gs3MhXF5rJbx9Y5WbKzn58fVCpVnVaS3NzcOq0pkqVLl2LgwIH4v//7PwBA9+7d4ebmhkGDBuHvf/87goOD6xyj0Wig0bAZnppuw9HLAID74tpyRtgWcH0tGgAsX74c27Ztw8qVK7F06dI6+y9fvtzs/6+++io2b96MLVu2IC4uzrTP3XffjUWLFgEw/mjZtWsXli9fjrVr1zbvCVGTSa0S0YG2Mfna9aRunfM5JTf9UU3ys+iqrVarER8fj+3bt5tt3759OwYMGFDvMeXl5VAqzZ9GpTJOysPhXNQcrpRVYceZXABgIWwLaEot2o0MBgNKSkrg43NtuHdiYmKdxxw5cmSDj8laNdtgGqkTYHstJ+G+bnBWKVBWpUdGYYXc4dBNWPyTcsGCBfj3v/+NTz75BKdOncL8+fORlpZm6qZZtGgRpk6datp/7Nix2LhxI1auXImUlBTs27cPzzzzDPr27YuQkBDrnQlRrS3HM1GtF+ja1tMmm5YdTVNq0W70xhtvoKysDBMnTjRty87OtugxWasmvxq9ASl5ZQBsa6SOxFmlRJSfsfXkLItibZpF3ToAMGnSJBQUFGDJkiXIyspC165dsXXrVoSHhwMAsrKyzPqZp0+fjpKSErz33nt49tln4e3tjWHDhmHZsmXWOwui63xdWwh7XxxbTVqSJbVo11u7di1eeuklbN68GQEBAU1+zEWLFmHBggWm/xcXFzNBaWGXrpSjSm+Ai7MK7dq4yB1OvaID3XEmpwRnc0oxLJYrlNsqi5MTAJgzZw7mzJlT731r1qyps+3pp5/G008/3ZSnIrJIWkE5jqUVQqkAxnavW89E1teUWjTJ+vXrMXPmTHz55Ze46667zO4LCgqy6DFZqyY/aYhudKA7lDY2UkcSE+iBb5HFlhMbx0pBcijfHDe2mgxo74cAT63M0bQOTalFA4wtJtOnT8fnn39e77xHCQkJdR7zxx9/vOljkrykyc2ibbDeRBJd2910jhOx2bQmtZwQ2SIhBDYnGRf5G9eT9UwtacGCBZgyZQp69+6NhIQEfPTRR3Vq0TIyMvDpp58CMCYmU6dOxdtvv43+/fubWkhcXFzg5eUFAJg7dy7uvPNOLFu2DOPHj8fmzZvx008/Ye/evfKcJN3S2RzbHakjMU3EllsCg0HYbAtPa8eWE3IYp7NLcC63FGonJe7pGiR3OK3KpEmTsHz5cixZsgQ9e/bE7t27b1qL9uGHH6KmpgZPPvkkgoODTbe5c+ea9hkwYADWrVuH1atXo3v37lizZg3Wr1+Pfv36tfj5UeNcm7bedpOTMB9XqFVKVFYbOGLHhrHlhBzGN8eNrSbDYgLgqXWWOZrWx5JatJ07dzbqMSdMmIAJEybcZmTUEvQGgZR840idDv62263jpFIi0s8NZ3JKcD63FKE2NsU+GbHlhByCEALf/ZYFABjbg106RC0t/Uo5qmoM0Dor0dZGR+pIOgTUTsbGmWJtFpMTcgi/ZxQj7Uo5tM5KDI31lzscolZHmrY+ys/21tS5kZScnMvliB1bxeSEHMJ3J4ytJsNiA+CqZm8lUUuTvuhtuRhWwpYT28fkhOyeEAJba5OT0d04twmRHKQv+g7+tp+cRJtG7JRyGRUbxeSE7N7JzGtdOsNiA259ABFZnZSc2EPLSaSfG5QKoKSyBnklOrnDoXowOSG7J3XpDI1hlw6RHIQQ11pOAmw/OdE4qRDu6wbgWq0M2RYmJ2TXhBD44XfjBF6j2KVDJIvMokqUV+nhpFSYvvRtXXt/1p3YMiYnZNfO5ZbiYn4Z1ColhsZwlA6RHKQv+Eg/Nzir7ONrJTqQI3ZsmX38FRE1YFttq8nADr7w4MRrRLKQFvyzhy4dSQe2nNg0Jidk17YlG5OTkV04XT2RXEzFsHaUnEgtJ0xObBOTE7Jbl6+W4/eMYigVwF2dA+UOh6jVkr7g29tRciLVnOSXVqGwvErmaOhGTE7Ibv14MgcA0DvcB37uGpmjIWq9LuTVJid2MMeJxE3jhGAvLYBr8ZPtYHJCduvH2i6dEV3YakIkl4JSHa6WVwOwr+QE4EyxtozJCdmlovJqHE69CgAY0Zn1JkRyuZBnXIm4rbcLXNQqmaOxjJRMSedAtoPJCdmlnWdzoTcIRAe4I8yXS54TycWeJl+7UXu2nNgsJidkl346lQuAhbBEcrPHehNJe3/jhHGsObE9TE7I7lTrDdh5pjY56cS1dIjkZM8tJ1LM6VfKUVmtlzkauh6TE7I7hy9eQUllDXzc1OgZ2kbucIhatWstJ/Yxbf31/N018NA6wSCA1ALWndgSJidkd6QunWGxAVApFTJHQ9R6VVTpkVFYAcA+W04UCgVH7NgoJidkd345bZzfhF06RPJKyS+FEIC3qzN83NRyh9MkphE7uWw5sSVMTsiuXMwvQ2pBOZxVCgzs4Cd3OEStmqnexN8dCoV9tmKaWk5YFGtTmJyQXdlx2til0yfChwv9EclMmh/EHrt0JB1MLSdMTmwJkxOyKztqR+kMjWGXDpHc7HkYsUSa6yQlvxQGg5A5GpIwOSG7UV5Vg4MpVwAAQ2OZnBDJ7YJpwT/7G6kjCW3jArVKicpqg6m4l+TH5ITsxv7zBajSGxDq42KXwxaJHIneIHAxv7Zbx99D5miazkmlRISfcZZpTsZmO5ickN24vkvHXovviBxFZmEFdDUGqJ2UaNvGRe5wbgvX2LE9TE7ILgghsPNMHgBgSIy/zNEQkTS6JcrPze7nG7qWnLDlxFYwOSG7cCGvDBmFFVA7KZEQxSHERHIz1ZvYcTGsRKqZ4Ygd28HkhOzCrrPGVpN+kT52tyw7kSOSukAcof6L3Tq2h8kJ2YXdtcnJndHs0iGyBaZhxHY8x4kkqjY5yS/Voai8WuZoCGByQnagslqPAykFAIDBrDchsgkpDjDHicRd44QgTy0A4EI+u3ZsAZMTsnmHLl6BrsaAIE8toh3gVxqRvSssr0J+aRUAINLP/rt1ANad2BomJ2TzTF06Hf04hJjIBki1GSFeWrhpnGSOxjpYd2JbmJyQzZOKYQd35KywRLbAkepNJBxObFuYnJBNyyqqwLncUigVwMAOvnKHQ0RwjDV1bsTkxLYwOSGbtvdcPgCgWztveLuqZY6GiADgQq7jDCOWSDUnaQXlqNYbZI6GmJyQTdt73picDOrAideIbIU0UifKgVpOgjy1cFWrUGMQuFRQLnc4rR6TE7JZBoPAvtrk5I5oJidEtqCqxoBLV4xf3h0cqOZEoVCYunZS2LUjOyYnZLNOZ5cgv7QKLs4qxIV5yx0OEQFIu1IGvUHAXeOEAA+N3OFYldRNxRE78mNyQjZr7/naKeujfKBx4pT1RLbgfG29SZS/m8MN7Y9iUazNYHJCNmtPbTHsHaw3IbIZjjhSR8IRO7aDyQnZpMpqPQ6nXgEADOJ6OkQ241py4jgjdSTXzxIrhJA5mtaNyQnZpF/TrqKy2gB/Dw06BjreLzQie3VtNWLH+1xG+LpBoQCKK2tM0/OTPJickE3af9640N+A9r4O169NZK+EEEjJdbzZYSVaZxVC27gCYNeO3JickE3ad8FYbzKQ9SZENiOvRIcSXQ2UCiDc11XucJrFtRE7TE7kxOSEbE5JZTV+u1wEgMkJkS05X/uFHebj6rAj6ExFsbkcTiwnJidkcw6mXIHeIBDh64q23i5yh0NEtRy53kQidVex5UReTE7I5khdOgPYakJkUy44cL2JhMOJbQOTE7I5UjHswPZMTohsiSMPI5ZI55ZRWIGKKr3M0bReTE7IpuSV6HAmpwQAkNDeV+ZoiOh6KXnS7LCO23Li46aGt6szhABS8tl6IhcmJ2RTDqQYW006BXvCx00tczREJCmvqkFGYQUAoIMDJyfmCwCyKFYuTE7Ipuy/cG1+EyKyHdIXtY+bGm0c/IcDhxPLj8kJ2RSp5SQhiskJkS1pDfUmkmtFsWw5kUuTkpMVK1YgMjISWq0W8fHx2LNnz0331+l0eP755xEeHg6NRoP27dvjk08+aVLA5LiyiipwMb8MSgXQN8pH7nCI6DrSSJ0ODjxSRyKd4/lctpzIxcnSA9avX4958+ZhxYoVGDhwID788EOMGjUKycnJCAsLq/eYiRMnIicnB6tWrUKHDh2Qm5uLmpqa2w6eHEtibZdOt7Ze8NQ6yxwNEV2vNcxxIrlWc1IKg0FAqeQSGi3N4uTkzTffxMyZMzFr1iwAwPLly7Ft2zasXLkSS5curbP/Dz/8gF27diElJQU+PsZfwxEREbcXNTkkKTnpz3oTIptzrVvH8ZOTdm1coFYpoasxIKOwAqE+jjlVvy2zqFunqqoKR48exYgRI8y2jxgxAvv376/3mG+++Qa9e/fG66+/jrZt26Jjx4547rnnUFFR0eDz6HQ6FBcXm93I8SWy3oTIJukNAin5raflxEmlRIQfFwCUk0XJSX5+PvR6PQIDA822BwYGIjs7u95jUlJSsHfvXvz+++/YtGkTli9fjq+++gpPPvlkg8+zdOlSeHl5mW6hoaGWhEl2KP1KOS5frYCTUoE+Eaw3IbIll6+Wo6rGALWTEm3btI4lJaQkjHUn8mhSQeyNS9gLIRpc1t5gMEChUOCzzz5D3759MXr0aLz55ptYs2ZNg60nixYtQlFRkemWnp7elDDJjkitJj1CveGmsbi3kWyAJYXyWVlZmDx5MmJiYqBUKjFv3rw6+6xZswYKhaLOrbKyshnPguojtR5E+blB1UrqLzoEcMSOnCxKTvz8/KBSqeq0kuTm5tZpTZEEBwejbdu28PLyMm3r1KkThBC4fPlyvcdoNBp4enqa3cixSUOI+3OUjl2SCuWff/55HDt2DIMGDcKoUaOQlpZW7/46nQ7+/v54/vnn0aNHjwYf19PTE1lZWWY3rVbbXKdBDZBW6HXkNXVuxDV25GVRcqJWqxEfH4/t27ebbd++fTsGDBhQ7zEDBw5EZmYmSkuvvcFnz56FUqlEu3btmhAyORohBA6mXAEA9Ge9iV26vlC+U6dOWL58OUJDQ7Fy5cp694+IiMDbb7+NqVOnmv1wuZFCoUBQUJDZjVpeayqGlVw/YodansXdOgsWLMC///1vfPLJJzh16hTmz5+PtLQ0zJ49G4CxS2bq1Kmm/SdPngxfX188+uijSE5Oxu7du/F///d/mDFjBlxcWkffJd3c5asVyCg01pvEh7eROxyyUFMK5RurtLQU4eHhaNeuHe69914cO3aswX1ZSN98pLqL1jABmySq9lzzS6tQWF4lczStj8XJyaRJk7B8+XIsWbIEPXv2xO7du7F161aEh4cDMPYlX9+U6+7uju3bt6OwsBC9e/fGww8/jLFjx+Kdd96x3lmQXZOGEPcI9YarmvUm9qYphfKNERsbizVr1uCbb77B2rVrodVqMXDgQJw7d67e/VlI33xaY8uJm8YJIV7GLkQWxba8Jn0TzJkzB3PmzKn3vjVr1tTZFhsbW6criEjCKesdgyWF8o3Rv39/9O/f3/T/gQMHolevXnj33Xfr/XGzaNEiLFiwwPT/4uJiJihWUFCqw9XyaigUrSs5AYw1NplFlbiQV4reHEXYovgzlWQlhLiuGJbJiT1qSqF8UyiVSvTp06fBlhONRgONRmO15yMjqdWgrbcLXNQqmaNpWe393bHnXD5bTmTAhf9IVulXKpBZVAlnlQK9wr3lDoeaoCmF8k0hhEBSUhKCg4Ot9ph0a+fzWs+aOjfiGjvyYcsJyUpqNenejvUm9mzBggWYMmUKevfujYSEBHz00Ud1CuUzMjLw6aefmo5JSkoCYCx6zcvLQ1JSEtRqNTp37gwAePnll9G/f39ER0ejuLgY77zzDpKSkvD++++3+Pm1ZtIXc4dW1qUDXJeccMROi+O3AcnqwEXOb+IIJk2ahIKCAixZsgRZWVno2rXrTQvlASAuLs7076NHj+Lzzz9HeHg4UlNTAQCFhYV4/PHHkZ2dDS8vL8TFxWH37t3o27dvi50XXZuErDW3nFy+WoHKaj20zq2rW0tOTE5IVtL8Jv0iWW9i7ywtlBdC3PTx3nrrLbz11lvWCI1uwwVpGHErTE583dTwcnFGUUU1LuSVoktIw3PykHWx5oRkk36lHBmFFVBxfhMim1Smq0FGoXGZkdbYraNQKFh3IhMmJySbgxeNrSbd2npxPR0iG5RS26Xj66ZGGze1zNHIQ0rKLjA5aVFMTkg2BzmEmMimnc8rAdA6u3QkLIqVB5MTko3UctKPxbBENsk0UofJCbt1WhiTE5JFZmEF0q6UQ6kAerPehMgmteZhxBIpOUnNL0eN3iBzNK0HkxOSxaHaVpOubb3goXWWORoiqk9rHkYsaevtAq2zElV6A9KvVsgdTqvB5IRkcbB2fpN+kezSIbJF1XoDUvOZnCiVCkT5Gc//XE6JzNG0HkxOSBamehPOb0Jkky4VlKHGIOCmViG4dnXe1io6kEWxLY3JCbW43JJKpOSVQaEA+nClTyKbdC6ntt4k0OO2Vpd2BNFSUWwOk5OWwuSEWtzhi1cBALFBnvByZb0JkS06V1sMG92Ku3QkHQI8AFx7Taj5MTmhFsd6EyLbd47DiE1M3Tq5pTAYbr7sAlkHkxNqcdfW02FyQmSrpOJPtpwA4T6ucFYpUFGtN03nT82LyQm1qKtlVThTe9Hrw+SEyCbpDQIptSN1omu7NFozJ5XSNGKHRbEtg8kJtahDqcZWkw4B7vBz18gcDRHVJ/1KOapqDNA6K9G2jYvc4diEDoEsim1JTE6oRUmTr/VlqwmRzZLqTdr7u0OlbN0jdSRS99a5XM510hKYnFCLOnSR9SZEtk76Ama9yTXRHLHTopicUIspqazGycwiAGw5IbJlUtdFdCDrTSTR13XrCMERO82NyQm1mKOXrsIggDAfVwR7sR+byFZxGHFdEb5uUCkVKNHVIKdYJ3c4Do/JCbWYg+zSIbJ5BoMwrUbMbp1r1E5KRPi6AmDdSUtgckIthsWwRLYvo7ACFdV6qFVKhPm4yh2OTZHqTs5yxE6zY3JCLaKiSo/fLhcC4GJ/RLbsbO08RFH+bnBS8SvielLdCVcnbn78y6MWcSz9Kqr1AsFeWoT6sN6EyFZJrQIdWQxbh/SanGVy0uyYnFCLkKas7xPh0+pXOCWyZdIXb8dA1pvcSEpOznHETrNjckItwjS/SRTrTYhsmZSccBhxXZF+bnCqHbGTXVwpdzgOjckJNbuqGgN+TbsKgCN1iGyZ/rqROuzWqUvtpESEnxsA4Ew2u3aaE5MTana/XS6ErsYAXzc12vuzqZjIVqVfKYeuxgCNE0fqNKSjqSiWI3aaE5MTanYHrxtCzHoTItslrRjeIYBr6jSERbEtg8kJNTvOb0JkH86ZimHZpdMQU3LCNXaaFZMTalY1egOOXjLWmzA5IbJtZ01r6rD7tSEdTWvslMBg4Iid5sLkhJpVclYxSnU18NQ6ITbIU+5wiOgmTMOIA9hy0pBwXzc4qxQoq9Ijo7BC7nAcFpMTalZSl06fCB/2YRPZsBq9ASl5ZQCAmCAmJw1xVilNhf1cY6f5MDmhZnUghfUmRPYgtaAcVXoDXJxVaOvNWZxvRpoD5kw2606aC5MTajYGg8DhVGnyNa6nQ2TLpHk7Oga6Q8lWzpuKqa074Yid5sPkhJrNmZwSFFVUw1WtQpcQ1psQ2bIz2cUA2KXTGDG19XOnORFbs2FyQs1GqjeJD28DZ65uSmTTpC/aGBau31JsbQJ3IbcU1XqDzNE4Jn5jULM5eLEAAKesJ7IH0gRssWw5uaW23i5wU6tQpTcgNb9M7nAcEpMTahZCiOsW+2O9CZEtK6+qQdqVcgDs1mkMpVKBjrWvE7t2mgeTE2oWF/LKkF9aBY2TEt3beckdDhHdxNmcUggB+Lmr4eeukTscuyC1MHEBwObB5ISahdSlExfmDY2TSuZoiOhmWAxrOWkae7acNA8mJ9QsDprmN2GXDpGtMxXDBrIYtrGkRO5MTrHMkTgmJidkdUIIU8tJ/ygWwxLZOqlrgsWwjSctx5F+pQKluhqZo3E8TE7I6i4VlCOnWAe1SoleYW3kDoeIbkGaTIzdOo3n46aGv4exPuccJ2OzOiYnZHVSq0mPUC9onVlvQmTL8kt1yC+tgkJxrY6CGodFsc2HyQlZnVRv0o/1JkQ273SW8Ys13McVLmr+mLBEDItimw2TE7IqIQQOpNROvsZ6EyKbdyrLWNDZKZjFsJaSXrPkLBbFWhuTE7Kqy1crkFlUCSelAvHhrDchsnWnaocRx3LaeovFBhtbTk5lFUMIIXM0joXJCVmV1GrSvZ0XXNVOMkdDRLdyqrZbp1Mw600s1SHAHU5KBUoqa5BRWCF3OA6FyQlZ1UFOWU9kN6pqDDifKyUnbDmxlMZJhQ4B7gCuJXlkHUxOyKqklpP+TE6IbN6FvFJU6wU8tE5o18ZF7nDskpTUnWbdiVUxOSGrSb9SjstXK6BSKtCb9SZENs9UDBvkCYVCIXM09knqDpNqd8g6mJyQ1UhdOt3becFNw3oTIlt3baQO602aSmo5YbeOdTE5Iathlw6RfZHm52C9SdNJr11qQRnKqziNvbUwOSGrYXJCZF+klpNYJidN5ueugZ+7BkJwMjZralJysmLFCkRGRkKr1SI+Ph579uxp1HH79u2Dk5MTevbs2ZSnJRvGehMi+5JbUon80iooFddmOqWm6XTdfCdkHRYnJ+vXr8e8efPw/PPP49ixYxg0aBBGjRqFtLS0mx5XVFSEqVOnYvjw4U0OlmwX602I7ItUIxHh58Zp629TZ1PdCZMTa7E4OXnzzTcxc+ZMzJo1C506dcLy5csRGhqKlStX3vS4J554ApMnT0ZCQsItn0On06G4uNjsRrYt8YKxSyeBXTpEduFkZhEAoEuIl8yR2L/OIcbk5GQmv6usxaLkpKqqCkePHsWIESPMto8YMQL79+9v8LjVq1fjwoULWLx4caOeZ+nSpfDy8jLdQkNDLQmTWtj16+mw3oTIPkhfpF1CWG9yu6QE73RWCfQGTmNvDRYlJ/n5+dDr9QgMDDTbHhgYiOzs7HqPOXfuHBYuXIjPPvsMTk6Na+5ftGgRioqKTLf09HRLwqQWln6lAhmFFXBWKdA7gvUmRPbgZIbUcsLk5HZF+rnBxVmFimo9LuaXyh2OQ2hSQeyNk/UIIeqdwEev12Py5Ml4+eWX0bFjx0Y/vkajgaenp9mNbNf+C/kAgJ6h3lxPpxWzpFA+KysLkydPRkxMDJRKJebNm1fvfhs2bEDnzp2h0WjQuXNnbNq0qZmib11KKquRWlAOgN061qBSKkxFsb9nsGvHGixKTvz8/KBSqeq0kuTm5tZpTQGAkpISHDlyBE899RScnJzg5OSEJUuW4Pjx43BycsIvv/xye9GTTUhMYb1Ja2dpobxOp4O/vz+ef/559OjRo959EhMTMWnSJEyZMgXHjx/HlClTMHHiRBw8eLA5T6VVkIphQ7y08HFTyxyNY+ja1pjkSbU8dHssSk7UajXi4+Oxfft2s+3bt2/HgAED6uzv6emJEydOICkpyXSbPXs2YmJikJSUhH79+t1e9CQ7IQT2S8Ww7f1kjobkYmmhfEREBN5++21MnToVXl71/3Jfvnw57r77bixatAixsbFYtGgRhg8fjuXLlzfjmbQO0hdoZ7aaWE0XFsValcVt8AsWLMCUKVPQu3dvJCQk4KOPPkJaWhpmz54NwFgvkpGRgU8//RRKpRJdu3Y1Oz4gIABarbbOdrJPF/LKkFeig9pJibgwb7nDIRlIhfILFy40236rQvlbSUxMxPz58822jRw5ssHkRKfTQafTmf7PUX4Nk7oeWG9iPVL32MnM4gZLHajxLE5OJk2ahIKCAixZsgRZWVno2rUrtm7divDwcADGvuRbzXlCjiOxtt6kd3gbaJ05V0Jr1JRC+cbIzs626DGXLl2Kl19+ucnP15pcG0bM5MRaogPd4aRUoKiiGhmFFWjXxlXukOxakwpi58yZg9TUVOh0Ohw9ehR33nmn6b41a9Zg586dDR770ksvISkpqSlPSzaI9SYkaWyhfHM9Jkf5NU5ltR7nc40jSqQ6Cbp9GicVOtbOtMuundvHtXWoyQwGYZp8bUAHJietlaWF8o0VFBRk0WNylF/jnM0pQY1BoI2rM4K9tHKH41BMdScZLIq9XUxOqMmSs4pxtbwabmoVurfzljsckomlhfKNlZCQUOcxf/zxx9t6TLp+8jUv1kVYmZSc/M6Wk9vGSSmoyaT5TfpF+cJZxTy3NbOkUF4ide+WlpYiLy8PSUlJUKvV6Ny5MwBg7ty5uPPOO7Fs2TKMHz8emzdvxk8//YS9e/e2+Pk5kt8uG3/Vd2vHLh1r61b7I+1ERhGLYm8TkxNqsn3na7t02rNLp7VrSqF8XFyc6d9Hjx7F559/jvDwcKSmpgIABgwYgHXr1uFvf/sbXnjhBbRv3x7r16/nFAS36URGIQCgO+tNrK5zsCdUSgXySnTIKdYhiN1mTcbkhJqkqsaAQ7UrEQ/swPlNyFgoP2fOnHrvW7NmTZ1tQtx6DZIJEyZgwoQJtxsa1aqs1uNMtnECNracWJ+LWoXoAHeczi7Bb5cLEeQVJHdIdott8dQkSemFqKjWw9dNjZjaCnUism1nsktQrRfwcVOjrbeL3OE4pO61Sd8JFsXeFiYn1CT7zhvrTRLa+0KpZL8qkT2QvjC7tmUxbHOR6k6k2h5qGiYn1CRSMSy7dIjsx4naL0zWmzQf6bWVimKpaZickMVKdTU4llYIABjI9XSI7MZvGRyp09xigz3grFLgSlkVMgor5A7HbjE5IYsduFCAGoNAmI8rwnw5RTORPais1uNsjrEYtjuTk2ajcVIhJshYh3eCXTtNxuSELLa3tt7kjmi2mhDZi+SsYugNAn7uGgR5cohrc+rW1hvAtZYqshyTE7LYnnN5AIBBrDchshvSr/hubT1ZDNvMpJap3y4XyhuIHWNyQhbJKqrAhbwyKBXAANabENmNpPRCAECPUG9Z42gNTMlJehEMBhbFNgWTE7LInnPGLp3u7bzh5eosczRE1FhSctKTyUmziwn0gIuzCiW6GqTkl8odjl1ickIW2VubnAxivQmR3bhaVoWL+WUAmJy0BCeVEt1qhxRLIxvJMkxOqNEMBmGafO0O1psQ2Y2k2tqHSD83eLuq5Q2mlegZ5g3gWosVWYbJCTVaclYxCsqq4KZWIS6sjdzhEFEjJdX+emerScuRXmsmJ03D5IQabddZ4yidhPZ+UDvxT4fIXrDepOVJr/Xp7BJUVOnlDcYO8RuGGk1KTgbH+MscCRE1lhACx2u7dZictJxgLy0CPDTQGwR+z+R8J5ZickKNUlJZjV8vXQUADI5mckJkL1ILylFYXg21kxKdgj3lDqfVUCgUpmHbSSyKtRiTE2qU/bVT1kf4csp6InuSlG78UdElxJPdsS2MdSdNx79UapTdUpdOR7aaENmTYyyGlU1c7YidX9OuyhuIHWJyQrckhMDu2inr72RyQmRXjqQavxh7h/vIHEnr06OdN1RKBbKKKpHJFYotwuSEbiklvwzpVyqgVinRP8pX7nCIqJFKdTU4nV0MAIgP5/D/luamcUKnYOMKxUcvsfXEEkxO6JZ2nM4FAPSL8oGbxknmaIiosY6nF8IggLbeLgjy4krEcpBarJicWIbJCd3SjjPG5GRITIDMkRCRJaQuHbaayKdX7WvP5MQyTE7opkp1NTh08QoAYCjnNyGyK0drCzF7RzA5kUvv2uQkOasY5VU1MkdjP5ic0E3tO5+Par1AuK8rIv3c5A6HiBpJbxA4VvtrvReXm5BNiLcLgr200BsEhxRbgMkJ3ZRUbzI0JgAKhULmaIiosc7llqBEVwM3tQqxQR5yh9OqSd1qv7Jrp9GYnFCDhBCmepOhsaw3IbInUr1JzzBvOKl4qZeTlJwcYXLSaPyLpQadzCxGTrEOLs4q9IvkHAlE9uRIqrFWLJ7zm8ju+hE7eoOQORr7wOSEGvTzKWOryR3RftA6q2SOhogaSwiBg7WF7PxhIb9OwR5w1zihpPLavDN0c0xOqEE/ncoBANzViV06RPbk8tUKZBVVwkmpME2hTvJxUilNXTvS6Ee6OSYnVK+c4kqcyCiCQgEMiw2UOxwisoDUatK9nRdc1Zw40Rb0izK2YB1MYXLSGExOqF5Sl06Pdt7w99DIHA0RWeLQxQIAQN9ILjdhK6TutUOpVyAE605uhckJ1YtdOkT2y1RvEsV6E1vRra03tM5KXCmrwvncUrnDsXlMTqiOiio99p3PBwDc1ZldOkT2JLuoEpcKyqFUcNp6W6J2UpomwzvIupNbYnJCdew6mwddjQHt2rggJpCTNxHZk0O1Q4g7h3jCU+ssczR0vb5S1w6Tk1tickJ1/HgyGwAwonMQZ4UlsjMHU4z1Jv1Yb2JzpPfk4MUC1p3cApMTMlOtN5jqTUZ2YZcOkb1JvCAlJ6w3sTVxYd5QOymRU6xDSn6Z3OHYNCYnZOZgyhUUV9bA102N3hG8uBHZk6yiCqTkl0GpAPpFseXE1midVYivrTvZX1vXR/VjckJmttV26dzVKRAqJbt0iOzJvvPGVpNu7bzh5cJ6E1s0sIMxaZTeK6ofkxMyMRgEfkw2Jicju7JLh8jeSL/GB7Znq4mtGtDBDwCQmFLAdXZugskJmSRdLkROsQ5uahUGtPeTOxwisoAQAvsu1CYnHfj5tVXd23rBQ+OEoopqJGdynZ2GMDkhk62/ZQEAhnUK5EJ/RHbmQl4Zcop1UDspOb+JDXNSKU2T40nJJNXF5IQAGH91ff+7sUtnTLdgmaMhIkvtr/2iiw9rwx8XNi6htmV6H4tiG8TkhAAASemFyCisgJtahSEx/nKHQ0QWkr7opIJLsl3Se3Q49Qp0NXqZo7FNTE4IALD1BLt0iOxVtd6A/bWjP+6I5o8LWxcT6AF/Dw0qqw04knpV7nBsEpMTghACW0+wS4fIXiWlF6JEV4M2rs7o1tZL7nDoFhQKBe6sTSJ3n82TORrbxOSEcPxyEbt0iOzYrjPGL7hB0f6cn8hODK691u5iclIvJieEb5IyAQDD2aVDZJekL7g7O/LHhb0Y1MEPCgVwOrsE2UWVcodjc5ictHJ6g8CW34zJyfieITJHQ0SWyi/V4URGEQDgzmjOb2Iv2rip0b2dNwB27dSHyUkrdyClAHklOni7OmMQC+mI7M7ec8ZROp2DPRHgqZU5GrLE4I7s2mkIk5NWbnNSBgBgdLdgqJ3450Bkb3azS8duDe5obOnacy4PNXqDzNHYFn4btWKV1XrTxGvje7BLh8je6A0CO2uTk8FMTuxOj9oFGosra/BrWqHc4dgUJiet2M4zuSiprEGwlxZ9InzkDoeILHQs7SqulFXBU+uE3hGcst7eOKmUphGSP5/KkTka28LkpBX76qixS2dczxAoOfyQyO78dCoXADA0NgDOKl7O7dFdnYwrwP/E5MRMk/6aV6xYgcjISGi1WsTHx2PPnj0N7rtx40bcfffd8Pf3h6enJxISErBt27YmB0zWUVCqw84zxgvbhF7tZI6GiJpC+kIbXvsFR/ZncIw/nJQKXMgrQ2p+mdzh2AyLk5P169dj3rx5eP7553Hs2DEMGjQIo0aNQlpaWr377969G3fffTe2bt2Ko0ePYujQoRg7diyOHTt228FT021OykSNQaB7Oy9EB3rIHQ4RWehSQRnO55bCSalgvYkd89Q6o2+ksVudrSfXWJycvPnmm5g5cyZmzZqFTp06Yfny5QgNDcXKlSvr3X/58uX485//jD59+iA6OhqvvvoqoqOjsWXLltsOnppuw6+XAQAPsNWEyC5JXTp9I33g5eIsczR0O9i1U5dFyUlVVRWOHj2KESNGmG0fMWIE9u/f36jHMBgMKCkpgY9PwwWYOp0OxcXFZjeyntPZxTiZWQxnlQLjOEqHyC79lMwuHUchJSeHU6+iqLxa5mhsg0XJSX5+PvR6PQIDzT8MgYGByM7ObtRjvPHGGygrK8PEiRMb3Gfp0qXw8vIy3UJDQy0Jk25h/eF0AMCw2AC0cVPLHA0RWepKWRUOpV4BANzN5MTuhfm6IibQA3qDYOtJrSYVxCoU5iM7hBB1ttVn7dq1eOmll7B+/XoEBAQ0uN+iRYtQVFRkuqWnpzclTKpHZbUem44ZR+k82CdM5miIqCm2J2dDbxDoEuKJMF9XucMhK7inaxAAmOaeau0sSk78/PygUqnqtJLk5ubWaU250fr16zFz5kx88cUXuOuuu266r0ajgaenp9mNrGPbyWwUllcjxEvLGSXJqiwZxQcAu3btQnx8PLRaLaKiovDBBx+Y3b9mzRooFIo6t8pKLpImfYGNqv1CI/s3qpvxvdx9Lg+luhqZo5GfRcmJWq1GfHw8tm/fbrZ9+/btGDBgQIPHrV27FtOnT8fnn3+OMWPGNC1Ssoq1h4yjqv7YO5RLq5PVWDqK7+LFixg9ejQGDRqEY8eO4a9//SueeeYZbNiwwWw/T09PZGVlmd202ta9fkxRRTX2nTeup3NP12CZoyFriQn0QKSfG6pqDPjldK7c4cjO4m6dBQsW4N///jc++eQTnDp1CvPnz0daWhpmz54NwNglM3XqVNP+a9euxdSpU/HGG2+gf//+yM7ORnZ2NoqKiqx3FtQoF/PLcCDlChQKYGIf1vGQ9Vg6iu+DDz5AWFgYli9fjk6dOmHWrFmYMWMG/vWvf5ntp1AoEBQUZHZr7X4+lYNqvUDHQHd0CHCXOxyyEoVCYWoJ++H3LJmjkZ/FycmkSZOwfPlyLFmyBD179sTu3buxdetWhIeHAwCysrLMfi19+OGHqKmpwZNPPong4GDTbe7cudY7C2oUqdVkcEd/tPV2kTkachRNGcWXmJhYZ/+RI0fiyJEjqK6+NlqhtLQU4eHhaNeuHe69996bzo/UWkb5SV06bDVxPKNq39Mdp/NQUaWXORp5OTXloDlz5mDOnDn13rdmzRqz/+/cubMpT0FWVlGlN43SeaRfuMzRkCNpyii+7OzsevevqalBfn4+goODERsbizVr1qBbt24oLi7G22+/jYEDB+L48eOIjo6u85hLly7Fyy+/bL0Ts0HFldXYVbvQH+tNHE/Xtp5o18YFl69W4JfTuRjTvfUmoFyMoZXYcjwTRRXVaNfGBUNjGx4pRdRUlo7iq2//67f3798fjzzyCHr06IFBgwbhiy++QMeOHfHuu+/W+3itYZTftt+zUVVjQHSAO2KDOLOzo1EoFBhbO/fU5qQMmaORF5OTVkAIgf8kpgIAHukfzkJYsqqmjOILCgqqd38nJyf4+vrWe4xSqUSfPn1w7ty5eu9vDaP8vjmeCQAY3zOkUdM3kP0Z39OYnOw8k9eqJ2RjctIK/JpWiJOZxdA4KTGpNwthybqaMoovISGhzv4//vgjevfuDWfn+qdiF0IgKSkJwcGts6k7t6TSNEpnXI+2MkdDzSU2yBMxgR6o0hvww8nWWxjL5KQVWL3vIgBgbI8QzghLzcLSUXyzZ8/GpUuXsGDBApw6dQqffPIJVq1aheeee860z8svv4xt27YhJSUFSUlJmDlzJpKSkkyP2dp891sWDAKIC/PmxGsOblxPqWsnU+ZI5NOkgliyH5evlpuq+2cMjJQ5GnJUkyZNQkFBAZYsWYKsrCx07dr1pqP4IiMjsXXrVsyfPx/vv/8+QkJC8M477+CBBx4w7VNYWIjHH38c2dnZ8PLyQlxcHHbv3o2+ffu2+PnZgq9rv6j+0JOtJo5uXI8Q/HPbGSSmFCC7qBJBXq1vbh+FkKrQbFhxcTG8vLxQVFTkkP3IzemVb5Oxau9FDOzgi89m9Zc7HLJT9vgZtMeYG3I+txR3vbkLKqUCBxYNh7+HRu6QqJlNWLkfRy5dxcJRsZg9uL3c4TRZUz+H7NZxYEUV1VhXO7fJrEFRMkdDRE315VHjyKOhMf5MTFqJCfHtAABfHEmHHbQhWB2TEwe27lAayqr0iA5wxxCuo0Nkl6r1Bmw4ahxW+kcWtLcaY7oHw8VZhZS8MvyaVih3OC2OyYmDqqzW4997jYWwjw2K4rBDIju160we8kt18HVTYxjnKGo1PLTOGN3NODLtyyOON2fPrTA5cVBfHr2MvBIdQry0+EMcC+iI7JXUpXNfXFs4q3jJbk0m9jZ27Ww5nonyqta1UjH/0h1Qtd6AD3ZeAAA8Mbg91E58m4nsUW5JJX4+ZVyhll06rU/fSB9E+LqirEqPb39rXXOe8FvLAX19LAMZhRXwc9dgElcfJrJb6w+lo8YgEBfmjRhOV9/qKBQKTOoTBgD47MAlmaNpWUxOHEy13oD3dpwHAMwaFAmts0rmiIioKWr0BnxeO9puagIX62ytJvZuB7VKieOXi3A8vVDucFoMkxMHs+HoZVwqKIefuxpT+vOCRmSvfj6di6yiSvi4qTGqa+ucsp8AX3eNaXXi/7Wi1hMmJw5EV6PHOz8bF0X705AOcNNwAmAieyV9EU3sHcoW0Fbukdofmt8cz0RheZXM0bQMJicOZO3BNGQWVSLIU4uH+4XJHQ4RNdH53FLsOZcPhQL8LBN6hXmjc7AndDUGrD/cOoYVMzlxECWV1Xj3F2OtyVPDOvCXFpEdW7U3BQAwPDYQoT5c5K+1UygUmD4wAgCwel8qqmoM8gbUApicOIiVOy+goKwKUf5uHKFDZMfyS3XY8KtxRtgnBnPZCTIa3zME/h4aZBdX4rsTjr9aMZMTB5BRWIFVtbPBLhrViRM1Edmx/yZeQlWNAT1CvdE7vI3c4ZCN0DipMK121NbHuy86/Ho7/BZzAP/84TR0NQb0i/TBXZ04vTWRvaqs1uO/tYWwj3PZCbrBw/3C4eKsQnJWMfZfKJA7nGbF5MTOHUwpwNdJmVAogL+N6cyLGZEdW3soDVfKqtCujQtGdgmUOxyyMW3c1KYp7d+vnc/KUTE5sWPVegNe3HwSAPBgnzB0a+clc0RE1FSV1Xp8sMu47MScIR3gxO5Zqsfjg9vDWaXA/gsFOJJ6Re5wmg3/+u3Yp4mXcCanBG1cnfHnkTFyh0NEt+HLo5eRU6xDsJcWD8RzsU6qX1tvFzzQy9h68s4vjtt6wuTETmUWVuCt7WcBAH+5JxZt3NQyR0RETVVVY8DK2mb6Pw1pD40TpwKghs0Z0gEqpQK7z+bhWNpVucNpFkxO7JAQAn/ddAKluhrEh7fBRK5WSmTX1h82TqAY4KHh55luKczXFffFGVvX3qz9kepomJzYoU3HMrDzTB7UTkose6A7lEoWwRLZqzJdDd7+2dhq8jQnUKRGemZYNJxVCuw5l4/95/PlDsfqmJzYmeyiSry8JRkAMHd4NDoEuMscERHdjlV7LyK/VIdwX1c82JdT1VPjhPm64uF+xnlPlv1w2uHmPWFyYkcMBoHnvjyOoopqdG3ricfv5OyRRPasoFSHj3Ybp6p/dkQMJ1Akizw1rAPc1Cocv1yErSey5Q7HqvhJsCOr96di7/l8aJ2VWD4pjhcyIjv35vazKNXVoEuIJ+7tFix3OGRn/Nw1mDXI+CP1tR9OobJaL3NE1sNvNztx4nIRln1/GgDw/JjO7M4hsnO/ZxTh80NpAIAX7+3M2jFqkicGRyHIU4v0KxX4954UucOxGiYndqCovBpzPj+KKr0Bd3UKxCNcQp3Irgkh8PKWkxACGNsjBP2ifOUOieyUq9oJi0bHAgDe33EBWUUVMkdkHUxObJzBIPDcV8eRfqUCoT4ueOOPPThFPZGd25yUicOpV+HirMKiUbFyh0N2blyPEPSJaIOKaj3+/u0pucOxCiYnNm75z+ewPTkHapUSKybHw8vVWe6QiOg2XCmrwpJvjSPunhrWASHeLjJHRPZOoVDgpXFdoFIq8N2JLPyUnCN3SLeNyYkN23I8E+/8fA4A8I/7unLtHCIH8Pdvk3GlrAoxgR54bBBH3JF1dAnxwqxBkQCAFzb/jpLKapkjuj1MTmzU0UtX8NyXxwEAj98ZhT9y1kgiu7frbB42HsuAQgG89kA3qJ14CSbrmTe8I8J9XZFVVInXfzgjdzi3hZ8MG3Q2pwQz1hyBrsaA4bEB+Ms97JMmsneF5VX481fGHxzTEiIQF9ZG5ojI0bioVVh6XzcAwH8PXMKus3kyR9R0TE5sTPqVckz75BCKKqoRF+aN9yb3gopDDInsmhACz2/6HTnFOrT3d+MPDmo2Azr4YVqCcebY5748jitlVTJH1DRMTmxIRmEFHvr4ALKKKtHe3w2fTOsDFzXX2SCydxt+zcB3J7LgpFRg+aQ4fq6pWS0c1Qnt/d2QV6LDwg2/2eXU9kxObERGYQUe+ugALl+tQISvKz6b1R9t3NRyh0VEt+l0djH+9vUJAMD8uzuysJ2anYtahbcfjIOzSoEfk3Owau9FuUOyGJMTG3A+txQTVu5H2pVyhPm4Yu3j/RHkpZU7LCK6TSWV1fjT/35FZbUBd3b0x58Gt5c7JGolurb1wt/GdAYAvPb9aRxJvSJzRJZhciKzY2lXMfHDRGQVVaJDgDvWP9EfwV6c94DI3hkMAgu+OI6L+WUI8dJi+aSenKKeWtTUhHCM7RGCGoPAnM9+RXZRpdwhNRqTExl9+1smHvzoAK6UVaFbWy988UQCExMiB/H6tjPGCRSdlHj/4V7wYTcttTCFQoHX7u+GjoHuyC3RYdanh1FeVSN3WI3C5EQGeoPAm9vP4qnPj0FXY8Cw2ACsfbw/L15EDuKLI+n4YNcFAMA/J3TnsGGSjZvGCaum9YGPmxq/ZxRj/vok6A22XyDL5KSFXSmrwqNrDptmfn10YAQ+ntob7honmSMjImvYnpyDRRuNBbDPDOuA8T3byhwRtXahPq74aEo81Coltp3MwQubf7f5ETxMTlrQnnN5uGf5buw+mwetsxJvTeqBxWO7cB4TIgeReKEAT37+K/QGgQnx7TD/7o5yh0QEAOgd4YPlD/aEQgF8fjAN//rRtmeQZXLSAsp0NVi8+XdMWXUIuSU6dAhwx6Y5A3FfXDu5QyMiKzmQUoCZ/zmMqhoD7uoUiNfu78YVxMmmjO4WjH/8wTiD7Ps7LuDN7WdttgWFfQnN7JfTOXjh65PIKKwAADzSPwzPj+7MSZiIHMj+8/mY+Z8jqKjWY1C0H96bHAcnFX/7ke2Z3C8MZboa/GPrKbzz8znoDQY8NyLG5hJpJifN5GJ+Gf7+bTJ+Pp0LAGjXxgWv3d8dd0T7yRwZEVnT9yeyMHd9EqpqDBjc0R8fTomH1pk/Psh2PXZnFBQK4O/fncL7Oy6gsLwaS8Z3takSAyYnVpZbXIl3fjmHdYfSUWMQcFYpMGNgJJ4ZHg03Fr0SOZT/7E/FS1tOQghgROdAvPNQHBMTsguzBkVB46zCi5t/x2cH05BfqsNbk3rCVW0b31O2EYUDyCqqwIe7UrD2UBp0NQYAwLDYADw/phPa+7vLHB0RWVNVjQEvbTmJzw+mATA2lb9iY788iW5lSv9w+LmpMXddEradzMGElYn4eFpvtPWWf74tJie36cTlIqzam4Jvf8tCTe3Y8V5h3vjzPbHoH+Urc3REZG3ZRZV4eu2vOJx6FQoF8OeRsZg9OMrm+uyJGmNUt2D4eWgw+79HkZxVjHHv7sXbD8bJXoLA5KQJiiursfW3LKw9lIbjl4tM2/tF+uCZ4dEY0N6XFyoiB7TjdC4WfJGEq+XVcNc44e0He2J4p0C5wyK6LX0ifLD5qYF4/FNjgjLlk4N4amgHzB0eLVthN5OTRqqs1mPnmTx8+1smtifnmLpu1ColRnULwqw7orjaKJGDKtXV4B/fJWPtoXQAQJcQT7w3uRci/dxkjozIOtq1ccXGOQPw8paTWHsoHe/+ch67zubhjT/2QHSgR4vHw+TkJnKKK7H7bB5+PpWL3efyUF6lN93XIcAdE+Lb4Y/x7eDrrpExSiJqTj8l52DxN9emA3h0YAQWjoqFxomFr+RYtM4qLL2/OxLa++Fvm07gt8tFGPPuXjw1tAOeGBzVon/zTE6uk1+qw5HUqziQUoADKQU4nV1idn+IlxZjugfj3u4h6N7Oi103RA4sNb8M/9h6CtuTcwAYpwP454QeSGjPWjJybON6hKBvhA8WbvwNO8/k4c3tZ/H1sQy8cG9nDInxb5HvvlabnBRVVONUVjFOZhbjt8uFOJ5eiNSCcrN9FAqgW1svDIkJwIjOgegS4smEhMjB5ZXosGLnefzvwCVU6wWclArMGhSFZ4Z3sJlhlkTNLchLi9XT+2DLb1l45dtkpOSX4dE1hzGwgy/+ck8surfzbtbnd+hPmsEgkF1cidSCMlzML0NKXhku5JXibHYJMosq6z0mOsAd/aJ80D/KFwPa+3GlYKJWIre4Eqv2XsSniZdQUW3swh0S44/nR3eSpc+dSG4KhQLjeoRgSIw/3v/lPFbvS8W+8wUY994+3NUpAPPu6oiubZun1rJJycmKFSvwz3/+E1lZWejSpQuWL1+OQYMGNbj/rl27sGDBApw8eRIhISH485//jNmzZzc56PrU6A1455fzyCysQFZRBTKuViCjsALV+obXDWjr7YJOwZ7o0c4L3UO90bOdN7xcna0aFxHZtvO5pfh4dwo2HctAld5Y6N4j1BvPjeiIQdH+MkdHJD9PrTMWje6ER/qH463tZ/F1UgZ+OpWLK2VV2DhnYLM8p8XJyfr16zFv3jysWLECAwcOxIcffohRo0YhOTkZYWFhdfa/ePEiRo8ejcceewz/+9//sG/fPsyZMwf+/v544IEHrHISAKBSKvDvPSlmRasA4KRUINTHFeG+rmjv744ofzfEBHogOtADXi5MRIhauy3HM7H+iHEUTnx4Gzw5tD2GxgSwC5foBqE+rnhzUk88OawD3vn5HCbEN9/itQph4ZKE/fr1Q69evbBy5UrTtk6dOuEPf/gDli5dWmf/v/zlL/jmm29w6tQp07bZs2fj+PHjSExMbNRzFhcXw8vLC0VFRfD09Gxwvze3n4VapUCItwtCvF0Q6uOKQA8NF+Aiuk2N/QzaksbGfKWsCn/7+gRmDIxE7wifFoyQyPE19dphUctJVVUVjh49ioULF5ptHzFiBPbv31/vMYmJiRgxYoTZtpEjR2LVqlWorq6Gs3Pd1gudTgedTmf6f3FxcaPiW3B3x0btR0Qk8XFTY8XD8XKHQUTXsahJIT8/H3q9HoGB5jMiBgYGIjs7u95jsrOz692/pqYG+fn59R6zdOlSeHl5mW6hoaGWhElERER2rEn9HTf2xQohbto/W9/+9W2XLFq0CEVFRaZbenp6U8IkIiIiO2RRt46fnx9UKlWdVpLc3Nw6rSOSoKCgevd3cnKCr2/9kxlpNBpoNJx1lYiIqDWyqOVErVYjPj4e27dvN9u+fft2DBgwoN5jEhIS6uz/448/onfv3vXWmxCRfVqxYgUiIyOh1WoRHx+PPXv23HT/Xbt2IT4+HlqtFlFRUfjggw/q7LNhwwZ07twZGo0GnTt3xqZNm5orfCKyJcJC69atE87OzmLVqlUiOTlZzJs3T7i5uYnU1FQhhBALFy4UU6ZMMe2fkpIiXF1dxfz580VycrJYtWqVcHZ2Fl999VWjn7OoqEgAEEVFRZaGS0RWcKvPoHRd+Pjjj0VycrKYO3eucHNzE5cuXap3f+m6MHfuXJGcnCw+/vjjOteF/fv3C5VKJV599VVx6tQp8eqrrwonJydx4MABq8RMRM2vqZ9Di5MTIYR4//33RXh4uFCr1aJXr15i165dpvumTZsmBg8ebLb/zp07RVxcnFCr1SIiIkKsXLnSoufjRYZIXrf6DPbt21fMnj3bbFtsbKxYuHBhvfv/+c9/FrGxsWbbnnjiCdG/f3/T/ydOnCjuueces31GjhwpHnzwQavETETNr6mfwybNEDtnzhzMmTOn3vvWrFlTZ9vgwYPx66+/NuWpiMjGNdcUA4mJiZg/f36dfZYvX17vYzZ1CgIisj2cnYyIbktzTTHQ0D4NPSanICByHExOiMgqmmOKAUsek1MQEDkOh16VmIiaX3NNMdDQPg09JqcgIHIcbDkhotvSXFMMNLRPQ49JRI6DLSdEdNsWLFiAKVOmoHfv3khISMBHH32EtLQ0zJ49G4CxyyUjIwOffvopAOPin++99x4WLFiAxx57DImJiVi1ahXWrl1resy5c+fizjvvxLJlyzB+/Hhs3rwZP/30E/bu3SvLORJRy2FyQkS3bdKkSSgoKMCSJUuQlZWFrl27YuvWrQgPDwcAZGVlIS0tzbR/ZGQktm7divnz5+P9999HSEgI3nnnHTzwwAOmfQYMGIB169bhb3/7G1544QW0b98e69evR79+/Vr8/IioZSmEVIVmw4qKiuDt7Y309HS7Wa6dyJEUFxcjNDQUhYWF8PLykjucRuF1g0h+Tb122EXLSUlJCQBwaCCRzEpKSuwmOeF1g8h2WHrtsIuWE4PBgMzMTHh4eNx0aKKUoTnKLyWej+1ypHMBbn0+QgiUlJQgJCQESqV91NE39roBONb76UjnAvB8bF1zXTvsouVEqVSiXbt2jd7f09PTId50Cc/HdjnSuQA3Px97aTGRWHrdABzr/XSkcwF4PrbO2tcO+/gJRERERK0GkxMiIiKyKQ6VnGg0GixevNhhZonk+dguRzoXwPHOx1KOdP6OdC4Az8fWNdf52EVBLBEREbUeDtVyQkRERPaPyQkRERHZFCYnREREZFOYnBAREZFNYXJCRERENsXuk5OIiAgoFAqz28KFC296jBACL730EkJCQuDi4oIhQ4bg5MmTLRRx/VJTUzFz5kxERkbCxcUF7du3x+LFi1FVVXXT46ZPn17n/Pv3799CUZtbsWIFIiMjodVqER8fjz179tx0/127diE+Ph5arRZRUVH44IMPWijSm1u6dCn69OkDDw8PBAQE4A9/+APOnDlz02N27txZ531QKBQ4ffp0C0XdsJdeeqlOXEFBQTc9xlbfG2txlOsGwGuHLf198tphxfdG2Lnw8HCxZMkSkZWVZbqVlJTc9JjXXntNeHh4iA0bNogTJ06ISZMmieDgYFFcXNxCUdf1/fffi+nTp4tt27aJCxcuiM2bN4uAgADx7LPP3vS4adOmiXvuucfs/AsKCloo6mvWrVsnnJ2dxccffyySk5PF3LlzhZubm7h06VK9+6ekpAhXV1cxd+5ckZycLD7++GPh7OwsvvrqqxaOvK6RI0eK1atXi99//10kJSWJMWPGiLCwMFFaWtrgMTt27BAAxJkzZ8zei5qamhaMvH6LFy8WXbp0MYsrNze3wf1t+b2xFke5bgjBa4ct/X3y2mG998YhkpO33nqr0fsbDAYRFBQkXnvtNdO2yspK4eXlJT744INmiLDpXn/9dREZGXnTfaZNmybGjx/fMgHdRN++fcXs2bPNtsXGxoqFCxfWu/+f//xnERsba7btiSeeEP3792+2GJsqNzdXABC7du1qcB/pAnP16tWWC6yRFi9eLHr06NHo/e3pvWkqR75uCMFrh63gtaPp743dd+sAwLJly+Dr64uePXviH//4x02bMy9evIjs7GyMGDHCtE2j0WDw4MHYv39/S4TbaEVFRfDx8bnlfjt37kRAQAA6duyIxx57DLm5uS0Q3TVVVVU4evSo2WsKACNGjGjwNU1MTKyz/8iRI3HkyBFUV1c3W6xNUVRUBACNei/i4uIQHByM4cOHY8eOHc0dWqOdO3cOISEhiIyMxIMPPoiUlJQG97Wn9+Z2OOp1A+C1w1bw2tH098buk5O5c+di3bp12LFjB5566iksX74cc+bMaXD/7OxsAEBgYKDZ9sDAQNN9tuDChQt49913MXv27JvuN2rUKHz22Wf45Zdf8MYbb+Dw4cMYNmwYdDpdC0UK5OfnQ6/XW/SaZmdn17t/TU0N8vPzmy1WSwkhsGDBAtxxxx3o2rVrg/sFBwfjo48+woYNG7Bx40bExMRg+PDh2L17dwtGW79+/frh008/xbZt2/Dxxx8jOzsbAwYMQEFBQb3728t7czsc9boB8NphK3jtMGrye2NxW0sLWLx4sQBw09vhw4frPfarr74SAER+fn699+/bt08AEJmZmWbbZ82aJUaOHGkT55KRkSE6dOggZs6cafHzZWZmCmdnZ7FhwwZrncItZWRkCABi//79Ztv//ve/i5iYmHqPiY6OFq+++qrZtr179woAIisrq9litdScOXNEeHi4SE9Pt/jYe++9V4wdO7YZoro9paWlIjAwULzxxhv13m8v782NHOm6IQSvHbx2tO5rh5NlqUzLeOqpp/Dggw/edJ+IiIh6t0vV5ufPn4evr2+d+6VK4+zsbAQHB5u25+bm1sn4rMHSc8nMzMTQoUORkJCAjz76yOLnCw4ORnh4OM6dO2fxsU3l5+cHlUpV55fOzV7ToKCgevd3cnKq932Tw9NPP41vvvkGu3fvRrt27Sw+vn///vjf//7XDJHdHjc3N3Tr1q3BvxF7eG/q40jXDYDXDl47Wve1wyaTEz8/P/j5+TXp2GPHjgGA2QXkepGRkQgKCsL27dsRFxcHwNjvuWvXLixbtqxpAd+EJeeSkZGBoUOHIj4+HqtXr4ZSaXmvW0FBAdLT0xs8/+agVqsRHx+P7du347777jNt3759O8aPH1/vMQkJCdiyZYvZth9//BG9e/eGs7Nzs8Z7K0IIPP3009i0aRN27tyJyMjIJj3OsWPHWvR9aCydTodTp05h0KBB9d5vy+/NzTjSdQPgtYPXjlZ+7Whi645N2L9/v3jzzTfFsWPHREpKili/fr0ICQkR48aNM9svJiZGbNy40fT/1157TXh5eYmNGzeKEydOiIceekj2IYFSc+ywYcPE5cuXzYZuXe/6cykpKRHPPvus2L9/v7h48aLYsWOHSEhIEG3btm3xc5GGA65atUokJyeLefPmCTc3N5GamiqEEGLhwoViypQppv2lIWfz588XycnJYtWqVTYzHPBPf/qT8PLyEjt37jR7H8rLy0373Hg+b731lti0aZM4e/as+P3338XChQsFgBZtIm/Is88+K3bu3ClSUlLEgQMHxL333is8PDzs8r2xBke6bgjBa4ct/X3y2mG998auk5OjR4+Kfv36CS8vL6HVakVMTIxYvHixKCsrM9sPgFi9erXp/waDQSxevFgEBQUJjUYj7rzzTnHixIkWjt7c6tWrG+xXvt7151JeXi5GjBgh/P39hbOzswgLCxPTpk0TaWlpMpyBEO+//74IDw8XarVa9OrVy2z43LRp08TgwYPN9t+5c6eIi4sTarVaREREiJUrV7ZwxPVr6H24/m/oxvNZtmyZaN++vdBqtaJNmzbijjvuEN99913LB18PaT4OZ2dnERISIu6//35x8uRJ0/329N5YgyNdN4TgtcOW/j557bDee6MQQgjL2lqIiIiImo/dDyUmIiIix8LkhIiIiGwKkxMiIiKyKUxOiIiIyKYwOSEiIiKbwuSEiIiIbAqTEyIiIrIpTE6IiIjIpjA5ISIiIpvC5ISIiIhsCpMTIiIisin/D/wApiS8ggOkAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Sigmoid函数图像\n",
    "plt.subplot(121)\n",
    "plt.plot(X.detach(), sigmoid_y.detach())\n",
    "plt.title(\"Sigmoid Function\")\n",
    "# Sigmoid导函数图像\n",
    "plt.subplot(122)\n",
    "plt.plot(X.detach(), X.grad.detach())\n",
    "plt.title(\"Sigmoid Derivative function\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们发现，Sigmoid导函数最大值为0.25（在0点处取到），当x较大或者较小时，导函数取值趋于0。梯度消失或者梯度爆炸，始终是个概率问题。我们不能说导函数取值取值小就一定会发生梯度消失问题，只是导函数最大值越小，越有可能发生梯度消失。   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Sigmoid函数饱和区间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;一般来说我们会将靠近sigmoid函数的左右两端的区间称为函数的饱和区间（如下图圈出部分）（也就是自变量绝对值较大的区间），不难发现，当自变量落入饱和区间时，因变量会趋于0或者1，而无论自变量是极小（负数绝对值极大）还是极大，都会使得导函数取值趋于0，从而更容易导致模型梯度消失。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.loli.net/2021/02/24/nXN6Gq8521f7tDU.jpg\" alt=\"64\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "\n",
    "# 设置随机种子\n",
    "np.random.seed(0)\n",
    "torch.manual_seed(0)\n",
    "\n",
    "# 如果有GPU可用，使用GPU加速\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNetwork(nn.Module):\n",
    "    def __init__(self, layers):\n",
    "        super(SimpleNetwork, self).__init__()\n",
    "        self.layers = nn.ModuleList()\n",
    "        for i in range(len(layers) - 1):\n",
    "            self.layers.append(nn.Linear(layers[i], layers[i + 1]))\n",
    "            self.layers.append(nn.Sigmoid())\n",
    "\n",
    "    def forward(self, x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "\n",
    "# 定义一个深层的网络结构\n",
    "input_size = 1\n",
    "hidden_sizes = [64] * 10  # 十个隐藏层，每层64个节点\n",
    "output_size = 1\n",
    "layers = [input_size] + hidden_sizes + [output_size]\n",
    "model = SimpleNetwork(layers).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output shape: torch.Size([1, 1])\n"
     ]
    }
   ],
   "source": [
    "# 创建一个随机输入\n",
    "input_data = torch.randn(1, input_size).to(device)\n",
    "output = model(input_data)\n",
    "\n",
    "print(\"Output shape:\", output.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Loss: 0.002793394261971116\n"
     ]
    }
   ],
   "source": [
    "# 假设一个损失函数\n",
    "loss_fn = nn.MSELoss()\n",
    "target = torch.tensor([[0.5]]).to(device)  # 假设目标值为0.5\n",
    "loss = loss_fn(output, target)\n",
    "\n",
    "print(\"Initial Loss:\", loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Layer Weight Grad: 1.4004288180491375e-12\n",
      "Last Layer Weight Grad: 0.012725936248898506\n"
     ]
    }
   ],
   "source": [
    "# 清零梯度\n",
    "model.zero_grad()\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 获取第一层和最后一层的权重梯度\n",
    "first_layer_grad = list(model.parameters())[0].grad\n",
    "last_layer_grad = list(model.parameters())[-2].grad\n",
    "\n",
    "print(\"First Layer Weight Grad:\", first_layer_grad.abs().mean().item())\n",
    "print(\"Last Layer Weight Grad:\", last_layer_grad.abs().mean().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于Sigmoid激活函数叠加会造成严重梯度消失问题，因此复杂模型，如Sigmoid3和Sigmoid4的前几层在迭代过程中逐渐丧失变化的可能性，也就是学习能力，从而导致经过了很多轮的迭代，但最终结果只能和Sigmoid2和Sigmoid3持平的情况。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 需要注意的是，对于复杂模型来说，如果部分层失去学习能力（参数迭代的可能性），其实模型判别效力就和简单模型无异。从此也能看出对复杂模型进行有效训练的重要性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.tanh函数的梯度计算问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;如果说Sigmoid激活函数叠加更容易造成梯度消失，那tanh激活函数叠加是不是也是如此呢？首先我们来观察tanh激活函数导函数性质。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 绘制tanh函数的函数图像和导函数图像\n",
    "X = torch.arange(-5, 5, 0.1)\n",
    "X.requires_grad=True\n",
    "tanh_y = torch.tanh(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 反向传播\n",
    "tanh_y.sum().backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x25d2be14408>]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'tanh Function')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x25d2be7c908>]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'tanh Derivative function')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEICAYAAACwDehOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5IUlEQVR4nO3deXxc1Xn4/8+j0b5blmRL8oqxMWaxAbMFEiAsAZKGkGaB0ISkSfjRhjbJq01Jmy5k6bdka9KWACUJhbQBkhAIJiWsSUoIYLCJd2NblmVblqzd2teZ5/fHvSPG8oxmpNlnnvfrNfbMXeY+Mzozz5xzzz1HVBVjjDEGICfZARhjjEkdlhSMMcZMsaRgjDFmiiUFY4wxUywpGGOMmWJJwRhjzBRLCilIRFRETk52HJESkZ0icmmy4zAzS4VyFa+yIiJLRGRQRDxxeO6viUiXiByN9XOHOe69IvIPiTwmWFKICRFpFpErknTsO0Rkwv1A+G9/E8fjPSAiXwtcpqqnqepv43XMbJUi5WrAve0VkbtEpC6a541VWZn+3qjqIVUtVVVvtM897TiLgb8C1qjqwlg+97TjfFxEXgpcpqq3qupX43XMUCwpZIafuB8I/+0byQ7IZISfqGoZUAVcDywENs8lMYhIbqyDS5ClQLeqdiQ7kIRRVbtFcQP+G/ABI8Ag8Dfu8p8BR4E+4EXgtIB9HgC+B/wvMABsBFYErFfgVmAf0OtuKyGOfwfwP+GWA8vc5811H/8W+CrwezeGZ4HqgO0vBl4GjgGHgY8DtwATwLj7Wp90t20GrnDvFwDfBVrd23eBAnfdpUALzi+vDqAN+ESy/4apeEvFcgV4gK3AtwKWvQfY4paTl4EzA9Y1A7cD24AxINdfVoB697VVBWx/FtAF5AErgF8D3e6yHwOVod6bwPIN3ABsmhb754ENAWX0W8AhoB24FygK8h5c4R7D5x7nAX8ZnrZdYPm/A/gp8CP3b7ATWB+w7WLgMaDTfW13AacCo4DXPc6xgL/n1wL2/TTQCPQAG4D6ufxtw5a9ZBf+TLgFFoqAZX8KlPHWl+SWgHUPuH/Y89xC/GPgkWl/4F8ClcAStwBdHeLYdzD3pLAfWAUUuY/vdNctcQv0jTgf0PnAumAFNciH4ivAq0AtUIPzRfFVd92lwKS7TR5wLTAMzEv23zAVbylarr4CbHTvn42T3M/HSRg3uzEXBMS/BeeLsChIWfk18OmA5/4mcK97/2TgSvd11uAkwO+Gem84PikUu+V3ZcD614Eb3PvfxflSrXLfyyeBfwnxPlxKQBKY/jjIa7oD5wv+Wvc9+RfgVXedP6l+BygBCoGL3XUfB16a9rwP4H7WgHfiJMez3ffkP4AX5/K3DXez5qM4UdX7VXVAVcdwCspaEakI2OQxVX1NVSdxPrzrpj3Fnap6TFUPAb8Jsj7Qh0TkWMCtPsIw/0tV96rqCM6vG/8xbgKeV9WHVXVCVbtVdUuEz3kT8BVV7VDVTuDLwEcD1k+46ydU9SmcX0anRPjcWS/B5SqYVpwvU3B+uf6nqm5UVa+qPohTI7ggYPt/V9XDbhmb7iGcHx6IiOD8wn/IfZ2Nqvqcqo655ehfgUsiCVBVh4EnAp57JbAa2OAe59PA51W1R1UHgP/nHjtWXlLVp9Q5v/HfwFp3+Xk4NaQvqOqQqo6q6kshn+V4NwH3q+ob7t/+b4ELRWRZwDbR/m0BO6cQFyLiEZE7RWS/iPTj/JIAqA7YLLAnwzBQOu1pwq0P9FNVrQy4tUYYaqhjLMapRcxFPXAw4PFBd5lft/uFFey4ZgZJKFfBNODURsBpb/+rwB8kOGUn8O99eIbnehTni60eeAfOr93fAYhIrYg8IiJH3Nf6Pxz/OsOZSjjAR4BfuMmiBqcmsTkg5qfd5bEy/T0udM+pLAYOTiv/kTruc6WqgzjNTw0zHHdOnytLCrExfajZjwDX4bRJVuBUbQEkgTEN4RR+v9n0nDiM06YbTLhhdVtxviz8lrjLzOylVLkSkRzgj3C/uHHKyT9P+0FSrKoPB+wWsryo6jGcc1kfwnltD6vbFoLT7KI45yjKgT/h+NcZrhw+C1SLyDqc5PCQu7wL5zzBaQExV6hqpF+gx32u3C6wkSaUw8CSECfdZ/W5EpESnGbdIxEeO2KWFGKjHTgp4HEZTjW6G6cA/b8kxLQFeIfbf7sCp7oZqR8DV4jIh0QkV0Tmux8uOPG1Tvcw8PciUiMi1cA/4vzKM7OXEuVKRPJE5FScv+1CnKYcgO8Dt4rI+eIoEZF3i0jZLJ7+IeBjwB/z1hc3OK91EDgmIg3AF6btN2M5dH+NP4pznqIKeM5d7nPj/o6I1Lqvr0FE3hVhvHtxfvm/W0TygL/HaeOPxGs4nSvudN+rQhG5KOD1LBKR/BD7PgR8QkTWiUgBzt9+o6o2R3jsiFlSiI1/wfkiPCYif43T8+AgThbfhXPiNaFU9TngJzg9PzbjnISKdN9DOCfK/gqnqWALb7WL/hBY477WXwTZ/WvAJve424E33GVm9pJdrj4sIoM4PYs24CSjc/zNk6q6Cad9/i6cHi+NOCdMZ2MDsBJoV9WtAcu/jHNStQ+nN9Vj0/ab/t4E8xBOrepn05psbndjfdVtmnqeCM9rqWof8OfAD3D+DkM4Peoi2deLU9M6GafnUwvwYXf1r3F6Kh0Vka4g+74A/APwc5zEsoLYngeZIm/V1owxxmQ7qykYY4yZYknBGGPMFEsKxhhjplhSMMYYMyUtB6mqrq7WZcuWJTsMk6E2b97cpaqxvJgpYla2TTxFUrbTMiksW7aMTZs2JTsMk6FE5GD4reLDyraJp0jKtjUfGWOMmWJJwRhjzBRLCsYYY6ZYUjDGGDPFkoIxxpgpMUkKInK/iHSIyI4Q60VE/l1EGkVkm4icHbDuahHZ4677YiziMSbeoinzxqSyWNUUHgCunmH9NTgjIa7Emef3Hpgai/x77vo1wI0isiZGMRkTTw8whzJvTKqLyXUKqvritGnhprsO+JE7gcarIlIpInU4k4Q0qmoTgIg84m67KxZxmfQyPumjf3SCgdFJhsYmGR73MjLhZWzCy7jXx4TXx8SkMulTvD6f+7/iU8Wn4FNF3XnHfYpz3527xLnv3gE8OTl89oqVc451rmVeVdvmfFBznO7BMX6zp5P3n9VATk4i56/KbIm6eK2B46fla3GXBVt+frAnEJFbcH5xsWTJkvhEaeLK61MaOwZ582g/+zuHONwzzJFjI3T0j9I9OM7A2FxmKZw9ESjIjS4pRCBUmT8hKVjZnj2vT7ntoT/wSlM3XYNj3HpJqIkCzWwlKikES+M6w/ITF6reB9wHsH79epsEIk00dgzw7K52Xm7s5o1DvQyPewHni7m+ooj6ykLOWFTJ/JJ8qkryqSzOo7Qgl9KCXIrzcynKzyHf46EgL4d8Tw65HiHPk0OOCHkeQUTw5AgeEUSc5xWEHAERQdxjOcdM6K9JK9tx9MOXmnilqZvl1SV8+9k9XHxyNac3VCQ7rIyQqKTQgjNptd8inDlH80MsN2lsdMLLo5tbePi1Q+xs7Qdg9cIyPrR+MWsXV3BqXTnLq0soyPUkOdK4ClXmTZRUlf/6fTNvX1nNf9x4Fm//+m/48cZD/Mv7z0h2aBkhUUlhA3Cbe87gfKBPVdtEpBNYKSLLcaa2uwFnAm+Thia8Ph557RD/9sI+ugbHOa2+nH/6ozVcc3odCysKkx1eogUt80mOKSO09I7Q1jfKn126gsrifM5dXsVrB7qTHVbGiElSEJGHgUuBahFpAf4JyANQ1XuBp3Dm/G0EhoFPuOsmReQ24BnAA9yvqjtjEZNJrDeP9vPXP9vKjiP9nL+8irs+sorzl1cluskmYeZa5k30Nh7oAeC85VVT///6zQ46B8aoKStIZmgZIVa9j24Ms16Bz4RY9xTOB8ikqSe2HOFvHt1GaUEud990NtecvjBjk4FfNGXeROe1A91UFuexqrYMgPPd5PB6cw/XnlGXzNAygl3RbKJyz2/389lHtrB2cSVPf+4dXHtGXcYnBJNcGw/0cO6yqqluqKc3VFCU52FjkzUhxYIlBTNnd/+2ka8//SbvXVvPjz91vlXdTdy1949ysHt4qnYAkOfJ4Zyl86aalUx0LCmYOXliyxG+8fQerltXz3c+vI48jxUlE3+725zebGcuqjxu+ZmLKmjsGGTC60tCVJnFPslm1na39fOFR7dx3vIqvvGBM/HY1aQmQfZ3DgGwoqbkuOUrakqZ9CmHeoaTEVZGsaRgZmV0wstfPvwHygvzuPumszP9WgOTYpo6B6koyqOqJP+45Se5SaLJTRpm7iwpmFn5zvN72dcxyLc+eCbVpXYOwSTW/s5BTqopOaEzw0k1pYCTNEx0LCmYiDV2DPLD3x3gg+cs4tJTapMdjslCTZ1DrHATQKCKojyqSwvYb0khapYUTMS+8stdFOd7uP2a1ckOxWShgdEJOgbGppqKpjuppsSaj2LAkoKJyGsHenhxbyd/8c6V1mxkkqJp6iTziTUF//KmLksK0bKkYCLynef2Ul1awJ9csDTZoZgs5W8amt7zyG9FTQk9Q+P0Do0nMqyMY0nBhLWpuYdXmrr5s0tXUJRvvY1McjR1DuHJEZZUhUoKTg3CzitEx5KCCeu/Xm6mvDCXG89bHH5jY+LkUM8w9ZWF5OcG/9paOr8YgMO9dq1CNCwpmBm19Y3w9I6jfPjcxRTnJ2qkdWNOdOTYCA2VRSHX17vrjvSOJCqkjGRJwczooY2H8KnysQuXJTsUk+Vaj43QUFkccn1hnofq0nyOHBtNYFSZx5KCCcnnU36+uYVLVtWwuCr0h9GYeJvw+mjvH6WhcubJmuori2g9ZjWFaFhSMCG9eqCb1r5Rrj+rIdmhmCx3tG8Un0LDvNDNR+DM+33EkkJUYpIURORqEdkjIo0i8sUg678gIlvc2w4R8YpIlbuuWUS2u+s2xSIeExuPv3GE0oJcrlqzMNmhmCzn//VfP8M5BXCSRuuxEZw5jsxcRJ0URMQDfA+4BlgD3CgiawK3UdVvquo6VV0H/C3wf6oaOPj5Ze769dHGY2JjdMLLr3Yc5erTF1o3VJN0/l//M51oBidpDI97OTY8kYiwMlIsagrnAY2q2qSq48AjwHUzbH8j8HAMjmvi6OX9XQyOTfLuM216Q5N8EdcU/D2QrAlpzmKRFBqAwwGPW9xlJxCRYuBq4OcBixV4VkQ2i8gtoQ4iIreIyCYR2dTZ2RmDsM1Mnt3ZTmlBLm9bMT/ZoRjDkWOjVJfmU5g3c63VnxTsZPPcxSIpBJthJVSD3h8Bv5/WdHSRqp6N0/z0GRF5R7AdVfU+VV2vqutramqii9jMyOtTnt/dzmWra22+BJMSjhwbCVtLAKh3eydZTWHuYpEUWoDAS10XAa0htr2BaU1Hqtrq/t8BPI7THGWSaMvhXroGx7lqzYJkh2IM4L9GIXxSqCrJpzAvx2oKUYhFUngdWCkiy0UkH+eLf8P0jUSkArgEeCJgWYmIlPnvA1cBO2IQk4nCC7s7yM0RLjnFamQm+VSV1mMj1FWETwoiQn2ldUuNRtTjFqjqpIjcBjwDeID7VXWniNzqrr/X3fR64FlVDRzbdgHwuDuLUi7wkKo+HW1MJjovNXZx1pJKygvzkh2KMQyMTTI87p1qGgqnrqKQ9v6xOEeVuWIymI2qPgU8NW3ZvdMePwA8MG1ZE7A2FjGY2OgdGmf7kT4+f8WqZIdiDADtfc6wFbXlkSWFBWWFbDzQE35DE5Rd0WyO8/v9XajCxSurkx2KMQBTv/oXlEU2uVNteSEdA6N2AdscWVIwx3lpXxdlhbmc2VCR7FCMAaC936kpLIi0plBewIRX6bUL2ObEkoI5zu/3d3HhSfPJ9VjRMKmhfWC2ScHZzp9MzOzYJ99Maesb4XDPCOefZBesmdTR0T9GeWFuxMOtWFKIjiUFM+X15l4AzltWleRIjHlLe/9oxLUEcJqPwEkmZvYsKZgprx/ooSTfw6l1ZckOxZgpR2eZFGrcE9JHraYwJ5YUzJTXm3s4e+k8O58QgQiGi68QkSdFZKuI7BSRTyQjzkzQ0T9GbXlkPY8ACnI9VJXkW/PRHNmn3wDQNzLBnvYB1i+1pqNwIhkuHvgMsEtV1wKXAt92r/g3s+DzKR0Ds6spANSWFdgFbHNkScEA8IdDvajCucvmJTuUdBDJcPEKlIlzuX4p0ANMJjbM9Nc7PM6EVyO+RsFvgXutgpk9SwoGgK2H+xCBMxbZ9QkRiGS4+LuAU3EGh9wOfFZVfcGezIaFD23qwrVZ1hQWlBdY89EcWVIwAGxrOcbJNaWU2XhHkYhkuPh3AVuAemAdcJeIlAd7MhsWPjT/NQqRDnHht6C8kM6BMbw+u6p5tiwpGFSVrS3HWLu4MtmhpItIhov/BPCYOhqBA8DqBMWXMTrcX/u1s2w+qi0vxKfQNWjnFWbLkoLhyLERugbHWWtNR5GKZLj4Q8DlACKyADgFaEpolBmgc8D5Up9N7yN4K4n49zeRi8koqSa9bWvpA7CaQoQiHC7+q8ADIrIdp7npdlXtSlrQaapjYIyKorxZzwBYY0lhziwpGLYePka+J4fVC4M2eZsgwg0X784oeFWi48o0nQNjU1/ws1FTaklhrmLSfBTBhTyXikifiGxxb/8Y6b4m/na09rG6roz8XGtNNKmlc2Bs6gt+NqZqCnZOYdai/haI8EIegN+p6jr39pVZ7mviRFXZ2drPafVWSzCpp3NwbjWFwjwPZYW5VlOYg1j8NIzkQp547GtioK1vlGPDE6yps6RgUk/nwNisex751ZYVWFKYg1gkhUgu5AG40B0H5lcictos9zVxsqu1H4A19dbzyKSWIXdu5rnUFMBpQrKkMHuxSAqRXMjzBrDUHQfmP4BfzGJfZ0O76jMudrb2IwKrF9rIqCa1dLhf6HNPCjbUxVzEIimEvZBHVftVddC9/xSQJyLVkewb8Bx21Wcc7GrrY/n8EkoKrCOaSS2d0SaFUqspzEUskkLYC3lEZKE7MBgicp573O5I9jXxtbO1n1PtJLNJQVEnhbIChsa9DI3ZOISzEfXPwwgv5PkA8GciMgmMADeoqgJB9402JhOZ/tEJWnpHuPG8JckOxZgTdPrHPSqb3bhHfv4T1F2DY1YTnoWYvFMRXMhzF86okRHtaxJjX/sAYOcTTGrqHBwjN0eoLJrbII2BVzUvnV8Sy9Ayml2tlMX2HB0EYNUCSwom9XT0j1FdWkBOTrD+KOH5k0KHnVeYFUsKWWxv+wAl+R4aKouSHYoxJ5jrhWt+Nv7R3FhSyGJ7jg6wckHZnH+JGRNPnQNjVJfOfQbTquJ8PDliSWGWLClksb3tA5xiTUcmRXUNjs35JDNATo4wvyTf5lSYJUsKWaprcIzuoXFW2Ulmk4J8PqVrcJzqsrnXFACq7VqFWbOkkKX2HnV6HllNwaSi3uFxvD6d0wipgWrKCmyk1FmypJCl9rjdUVctKE1yJMacqGtwHIDqKE40g1NT6LKawqxYUshSjR2DlBfmRtW7w5h4mbqaOQY1ha7BcZxrZU0kLClkqcaOQU6uLcUdfcSYlOI/ORxtTaGmrIBxr4/+ERvqIlKWFLLU/k4nKRiTiqId98jP36W1c9BGS42UJYUsdGx4nK7BcUsKJmV1DY6Rn5tDWZRjFr11Adt4LMLKCpYUstD+Tmd4C0sKJlX552aOtnnTf07CeiBFzpJCFmrscJNCjXVHNampc3As6vMJ8FZNwXogRc6SQhZq7BgkPzeHhnk25pFJTf6aQrQqivLI84jVFGbBkkIWauwY5KTqEjw25pFJUV2DY9REeTUzgIjYVc2zZEkhC+3vHLLzCSZleX1Kz9B4TGoK4L9WwZJCpGKSFETkahHZIyKNIvLFIOtvEpFt7u1lEVkbsK5ZRLaLyBYR2RSLeExoY5NeWnqHOanGkoJJTd1DY/g0+msU/KymMDtRz7wmIh7ge8CVQAvwuohsUNVdAZsdAC5R1V4RuQa4Dzg/YP1lqtoVbSwmvEPdw/gUTqq2mahMaupyu49Wx6qmUFrAjiN9MXmubBCLmsJ5QKOqNqnqOPAIcF3gBqr6sqr2ug9fBRbF4LhmDpq6hgA4qcaSgklN/pPCtTGqKdSUFdA9NI7PZ0NdRCIWSaEBOBzwuMVdFsongV8FPFbgWRHZLCK3hNpJRG4RkU0isqmzszOqgLNZU6eTFJZZTcGkqFhdzexXXZqP16f0DtsFbJGIuvkICNaFJWhKFpHLcJLCxQGLL1LVVhGpBZ4TkTdV9cUTnlD1PpxmJ9avX28pf44OdA1SXVpAeeHcJkM3Jt78SSFmzUfuRD2dg2PMj9FzZrJY1BRagMUBjxcBrdM3EpEzgR8A16lqt3+5qra6/3cAj+M0R5k4aeocsqajGAjXucLd5lK3A8VOEfm/RMeYrjoHxijO91AS5RAXfjZX8+zEIim8DqwUkeUikg/cAGwI3EBElgCPAR9V1b0By0tEpMx/H7gK2BGDmEwIB7qG7CRzlAI6V1wDrAFuFJE107apBO4G3quqpwEfTHSc6cq5RiF2v+gtKcxO1KlYVSdF5DbgGcAD3K+qO0XkVnf9vcA/AvOBu92xTCZVdT2wAHjcXZYLPKSqT0cbkwmub3iC7qFxqylEb6pzBYCI+DtXBPa4+wjwmKoegqmasIlArK5m9rOkMDsxqZ+p6lPAU9OW3Rtw/1PAp4Ls1wSsnb7cxEdTlzPm0fJqu0YhSsE6V5w/bZtVQJ6I/BYoA/5NVX8U7MncDha3ACxZsiTmwaabzsExVsbw4sqSfA9FeR5LChGyK5qzSHO30/NoeXVxkiNJe5F0rsgFzgHeDbwL+AcRWRXsyVT1PlVdr6rra2pqYhtpGuociG3zkYjYXM2zEJszOSYtHOgcIkdgcZUlhShF0rmiBehS1SFgSERexKkV78WENDbppW9kIqbNR+B0S7WaQmSsppBFDnQP0zCviIJcT7JDSXdhO1cATwBvF5FcESnGaV7aneA4007XoHMtQaznDrfxjyJnNYUs0tw1xLL5dpI5WpF0rlDV3SLyNLAN8AE/UFXrWRdGrC9c86spK+C1Az0xfc5MZUkhS6gqzV1DXH/2TBebm0iF61zhPv4m8M1ExpXu4pYUSgvpHZ5gfNJHfq41kMzE3p0s0T00zsDYpNUUTErzN/HEo6YAzgisZmaWFLJEc5e/55ElBZO6/DWF+SXxSQp2sjk8SwpZ4kCXDYRnUl/HwCiVxXkxb+LxJ4WOfksK4VhSyBLN3UN4coRFNi+zSWEd/WMxGzI70FRSsJpCWJYUskRz9zCL5xWR57E/uUldHQNj1LqjmsaS/7qHjoHRmD93prFviCzR3DVkTUcm5XUOjFFbHvuaQn5uDlUl+VZTiIAlhSygqhzsHraeRyalqaqTFOJQUwBnJjc7pxCeJYUs0D00zuDYJEvn2/AWJnUdG55g3OuLyzkFcM4rdFrzUViWFLKAvzuq1RRMKvM37cSj+QigtqzQmo8iYEkhCzR3DwPWHdWkNv9J4Lg1H5UX0Dkwhs9ns/nOxJJCFjjodkdtqLTuqCZ1tbvt/fFqPqotK2DSp/QOj8fl+TNFTJJCuPlqxfHv7vptInJ2pPua6B3oGqKhssjGfDEpbaqmEMfmI+c41oQ0k6i/JSKZr9Zdt9K93QLcM4t9TZQOdg/bSWaT8jr6xygtyKU4Pz7jdPqTjSWFmcXip+PUfLWqOg7456sNdB3wI3W8ClSKSF2E+5ooqCrN3UM25pFJeU531PjUEuCtZqmOfuuBNJNYJIVg89VOH5851DaR7As489iKyCYR2dTZ2Rl10NmiZ2icgdFJlthsaybFdQyMxnx01EDWfBSZWCSFSOarDbVNJPs6C20e2znx9zyymoJJdR0DY9SWx6fnEUBRvoeyglwbKTWMWDTeRTpfbbBt8iPY10ThYLdzjcJSu0bBpDBVjdtgeIFqywtot+ajGcWiphDJfLUbgI+5vZAuAPpUtS3CfU0UmruHyRFYXGXdUU3q6h+dZGTCy8I41hQAFpQXWlIII+qaQiTz1eJMW3gt0AgMA5+Yad9oYzJvae4aor6yiIJcT7JDMSYk/xf1gor4JoWF5YVstLmaZxSTvl/h5qtVVQU+E+m+JnYOdg/Z8BYm5R3tc5JC3GsKFU5NwedTcnKCndI0djVThmvuHmZZtfU8MqntaH9iksLC8kImfUr3kF3VHIolhQx2bHicvpEJqymYlNfeF9+rmf0WuEnHziuEZkkhg/nnZbaeRybVHe0fZV5xHoV58T33tdA9Z+FvrjInsqSQwQ5OXaNgzUcmtbX3j079io8nf/PUUasphGRJIYMd6BpCBBbNs6RgUltb3+jUr/h4qi7NJ0espjATSwoZrLl7iPqKorhXyY2JVnv/aNxPMgPkenKoKSuwmsIMLClksOYuGwjPpL7xSR9dg+MJaT4CpwnJTjSHZkkhQ6kqB7qGrDuqSXn+eRQS0XwETg8kaz4KzZJChuodnqB/dNK6o5qU156gaxT8FlYUWvPRDCwpZCh/d1RrPjKp7mifM2ppopqPFpQXMjA6yfD4ZEKOl24sKWSoZjcpLLOkYFJcW98IAHUJaj6qryx0j2u1hWAsKWSoA11DeHKExdYdNS4inVtcRM4VEa+IfCCR8aWT1mOjFOd7qCzOS8jx6iuK3OOOJOR46caSQoY60D3EonlF5OfanzjWIp1b3N3u6zijAJsQWo+NUF9ZhEhiBqirr7SkMBP7xshQzV02OmocRTq3+F8APwc6EhlcumntG0lY0xE4J5pF4Mgxaz4KxpJCBvJ3R7WTzHETdm5xEWkArgfuJYxsn3+89dgoDZWJmwQqz5NDbVkBbVZTCMqSQgZq7x9jeNzLihpLCnESydzi3wVuV1VvuCfL5vnHRye8dA2OTTXpJEp9ZRGtfZYUgokqKYhIlYg8JyL73P/nBdlmsYj8RkR2i8hOEflswLo7ROSIiGxxb9dGE49xNHUNArC8ujTJkWSsSOYlXw88IiLNwAeAu0XkfQmJLo34LyJLSlKw5qOgoq0pfBF4QVVXAi+4j6ebBP5KVU8FLgA+M+2k3HdUdZ17sxnYYqCp0+mOepLVFOIl7NziqrpcVZep6jLgUeDPVfUXCY80xfl/rfu7iSZKQ2URrcdGcCaFNIGiTQrXAQ+69x8E3jd9A1VtU9U33PsDwG6mtb+a2DrQNURhXk7CrhDNNqo6CfjnFt8N/NQ/L7l/bnITGf+vdX830USpqyhkbNJHj83AdoJo52heoKpt4Hz5i0jtTBuLyDLgLGBjwOLbRORjwCacGkVviH1vAW4BWLJkSZRhZ7amzkGWV5faHLRxFG5e8mnLP56ImNKRv1toosY98nurW+oo80vjO9tbuglbUxCR50VkR5BbsC54Mz1PKU73vM+par+7+B5gBbAOaAO+HWr/bD4ZN1tNXUOcZD2PTBpoPTZCdWlBwod39/d2OmI9kE4QtqagqleEWici7SJS59YS6gjRH1tE8nASwo9V9bGA524P2Ob7wC9nE7w50fikj8M9w7x3bX2yQzEmrNa+URoSfD4B7AK2mUR7TmEDcLN7/2bgiekbiHOZ4g+B3ar6r9PW1QU8vB7YEWU8We9QzxA+tYHwTHpo6RmmYV5izycAzCvOoyjPQ0uvJYXpok0KdwJXisg+4Er3MSJSLyL+9taLgI8C7wzS9fQbIrJdRLYBlwGfjzKerNfY4fQ8WlFj3VFNavP5lJbekaSMzyUiLK4q4nDvcMKPneqiOtGsqt3A5UGWtwLXuvdfIvjFPqjqR6M5vjnR/k7nGoUVtZYUTGprHxhl3OtjcVVyBm1cUlXM4R5LCtPZFc0ZprFjkLqKQkoLou1YZkx8He5xmm6SlRQWzXOSgl2rcDxLChmmsWPQmo5MWjjk/kpfksSawtC4165VmMaSQgZRVfZ3DnKyNR2ZNHC4ZxiRxF/N7OevoRy2k83HsaSQQdr6Rp2B8CwpmDRwuHeYheWFFOQm9hoFv8VVTq8nO69wPEsKGaSxwznJfLI1H5k0cLhnOGnnE4CpXk+HLCkcx5JCBplKClZTMGngcE9yuqP6lRTkMr8knxbrlnocSwoZpLFzkIqiPKpL85MdijEzGp3wcrR/dKoJJ1kWVRVbTWEaSwoZZF/7AKsWlCZsrltj5sp/JXEyawrg9ECypHA8SwoZQlXZc3SAVQvKkh2KMWE1dzlX3i9P8pwfy+cXc6R3hLHJsBPkZQ1LChmivX+M/tFJTlloScGkvgNuUkj2aL7La0rwqfVACmRJIUPsaR8AsJqCSQtNXUPMK86jsji557/8U9b6Zys0lhQyxt6jlhRM+jjQNZgSI/kun+/E4K+5GEsKGWNP+wA1ZQVUlVjPI5P6DnQNTf1KT6aK4jzml+RbUghgSSFD7G0f4BSrJZg0MDQ2SXv/GCcl+SSz3/LqEposKUyxpJABvD5lX/sgKxck/5eXMeH4f5WnQvMRwEk1JVZTCBBVUhCRKhF5TkT2uf/PC7FdszuZzhYR2TTb/c3MmruHGJnwsqauPNmhGBNWqiWF5dWldA6MMTA6kexQUkK0NYUvAi+o6krgBfdxKJep6jpVXT/H/U0Iu1r7AVhTb0nBpD5/Ulg2P1WSgp1sDhRtUrgOeNC9/yDwvgTvb4Bdbf3keYSVtXZOwaS+xo5BGiqLKMpPzuio051c6yQF/6yF2S7apLBAVdsA3P9rQ2ynwLMisllEbpnD/mYGO1v7WVlbRn6unSIyqW9v+0BKXWS5dH4JeR5hz1FLChDBHM0i8jywMMiqL83iOBepaquI1ALPicibqvriLPbHTSa3ACxZsmQ2u2a8Xa39XHpKTbLDMCasCa+P/Z2DXJJC5TXPk8OKmlL2uheAZruwSUFVrwi1TkTaRaROVdtEpA7oCPEcre7/HSLyOHAe8CIQ0f7uvvcB9wGsX7/eJlV1dQyM0jU4xml2PsGkgeauISa8mnLdp1ctKGPzwd5kh5ESom1v2ADc7N6/GXhi+gYiUiIiZf77wFXAjkj3NzPb6T/JbD2PTBpI1eFYTllYxpFjIwyOTSY7lKSLNincCVwpIvuAK93HiEi9iDzlbrMAeElEtgKvAf+rqk/PtL+J3M4jfQCcajUFkwb2tg+SI6k3EdRKN5591oQUvvloJqraDVweZHkrcK17vwlYO5v9TeS2HO5jRU0J5YV5yQ7FmLD2Hh1g2fwSCvNSo+eRn//E9972Ac5akt2XS1l3lTSmqmw5fIy1iyuTHYoxEdnbnppzfiyeV0xhXo71QMKSQlpr63NOMq9dVJnsULKOiFwtIntEpFFETrjoUkRuEpFt7u1lEQlaW84mw+OTNHcPsSqFuqP65eQIpywoY3dbf7JDSTpLCmls6+FjAFZTSDAR8QDfA64B1gA3isiaaZsdAC5R1TOBr+L2nMtmu1r78Smc2VCR7FCCOr2hgh2tfahmd+dGSwppbGtLH3ke4dS61PvlleHOAxpVtUlVx4FHcK7On6KqL6uqv4/jq8CiBMeYcra7nSLOWJSaSeGMhgoGRic52J3ds7BZUkhjWw73cmpdOQW5qXXSLgs0AIcDHre4y0L5JPCrUCtF5BYR2SQimzo7O2MUYurZfqSPmrICFpQXJjuUoE53azD+5JWtLCmkqQmvj62H+zg7y3tKJIkEWRa0zUFELsNJCreHejJVvU9V16vq+pqa1LnSN9Z2HOnjjBRtOgLn2ol8Tw47LCmYdLTjSB8jE17OXVaV7FCyUQuwOODxIqB1+kYicibwA+A6t/t11hoen6SxY3Dq13gqys/NYXVdmdUUkh2AmZvXm3sAOHe51RSS4HVgpYgsF5F84Aacq/OniMgS4DHgo6q6NwkxppTdbc5J5lSuKYB7svlIdp9stqSQpl5v7mXZ/GJqy1KzfTaTqeokcBvwDLAb+Kmq7hSRW0XkVnezfwTmA3dPn1wqG/3h0DEAzkzRk8x+axdV0D86yf7O7J1bIaormk1y+HzKpuYeLj91QbJDyVqq+hTw1LRl9wbc/xTwqUTHlao2H+xlcVVRyp5k9jtnqdMcu/lgT8oNxZEoVlNIQ42dg/QOT3DuMms6MqlPVXm9uZdzl6b++a8VNSXMK87j9ebsHTHVkkIaemlfFwBvW1Gd5EiMCe9g9zBdg2OsT4NOESLC+mVVbHLP2WUjSwpp6KXGLpbNL2ZxVXGyQzEmrKlOEWlSsz132Tyau4fpGBhNdihJYUkhzYxP+ni1qZuLV1otwaSHTc29VBbnsaImPdro/TWazVnahGRJIc28caiX4XEvb1+ZuRc5mcyhqrzc1MX6pVXk5AS75i/1nF5fQVGeh5f3Z+elJZYU0sxL+7rw5AgXrpif7FCMCau5e5jDPSNcsip9arb5uTlcuGI+L+7L3CFHZhJVUhCRKhF5TkT2uf+f0GgoIqe4/bT9t34R+Zy77g4RORKw7tpo4skGz+9u55wl82xSHZMWXtzrfLG+Y1V61WzfsbKag93DHOzOvusVoq0pfBF4QVVXAi+4j4+jqntUdZ2qrgPOAYaBxwM2+Y5/vdv324RwqHuYN48OcNVpdn2CSQ8v7u1kSVUxS+eXJDuUWXm7m8RedHv6ZZNok8J1wIPu/QeB94XZ/nJgv6oejPK4WenZXUcBuGrNwiRHYkx445M+Xmnq5h1p1HTkd1J1CQ2VRVM1nWwSbVJYoKptAO7/tWG2vwF4eNqy29zZqe4P1vzkly3DC8/k2V3trF5YxpL51hXVpL5XmroZHvdy6apwXwupR0S4bHUNL+3rYmTcm+xwEipsUhCR50VkR5DbdeH2nfY8+cB7gZ8FLL4HWAGsA9qAb4faP1uGFw6lo3+UTc09XHWa1RJMevjV9jZKC3LTtvv0NafXMTLh5f/2diQ7lIQKO/aRql4Rap2ItItInaq2iUgdMNO7dw3whqq2Bzz31H0R+T7wy8jCzj5PbGnFp/C+dfXJDsWYsCa8Pp7ZeZTLT62lMC89J4E6f3kV84rzeGr7Ua4+vS7Z4SRMtM1HG4Cb3fs3A0/MsO2NTGs6chOJ3/XAjijjyVg/f6OFtYsrOSlNLgAy2W1jUw+9wxNck8ZfprmeHN512kJe2N3O6ET2NCFFmxTuBK4UkX3Ale5jRKReRKZ6EolIsbv+sWn7f0NEtovINuAy4PNRxpORdrf18+bRAd5/1kwzPhqTOjZsPUJxvodLT0nvpt5rz6hjaNzLC7uzpwkpqqGz3dmkLg+yvBW4NuDxMM7Y8tO3+2g0x88WD792iHxPDu85M31/dZnsMTA6wZNb23jv2vq0bTryu+jkauoqCnnk9UO8O0s+f3ZFc4obGJ3g55tbeM+ZdcwvLUh2OMaEtWFrKyMTXm44b3H4jVOcJ0f44PrFvNTYxeGe4WSHkxCWFFLco5tbGBr3cvPbliU7FGPCUlUefu0QqxeWsW5xZbLDiYkPrV8EwE83HU5yJIlhSSGFTXh93P/7A5y1pJK1GfIBM5ntlaZudhzp56YLliKSHgPghbNoXjGXr67lf149yPD4ZLLDiTtLCinssTdaONwzwm2XnZzsUIyJyD2/3U91aQEfPGdRskOJqT+7dAW9wxM88lrm1xYsKaSo8Ukf//HrRtYuquCdq9PvilCTfba1HON3+7r45MXL0/4E83TnLK3ivOVVfP93TRnfPdWSQop64OUDtPSO8LkrV2VMNdxkLlXln/93N1Ul+fzJBUuSHU5cfPbylbT1jfLAy83JDiWuLCmkoPb+Uf7t+X28c3Utl51itQST+p7ZeZSNB3r4/JWrKMvQYd0vOrmay1fXctevG+kcGEt2OHFjSSHFqCr/8IsdTPiUf/qjNckOx5iw+kYm+MqTu1i1oJQbz03/bqgz+bt3n8rohJcvP7kz2aHEjSWFFPPwa4d5dlc7X7jqlLQbg95kpy8/uZP2gTG+8YG15Hoy+ytlRU0pn7tiJb/c1saTW1uTHU5cZPZfMM1sOXyMLz+5k4tPruaTFy9PdjjGhPWT1w/x2BtH+MylKzLmuoRwbr3Eea1/99h2GjsGkh1OzFlSSBGHuof51IObqC0v4Ls3rEubSc5N9nplfzd//4sdvH1lNX95+cpkh5MwuZ4c7vrIWRTk5fDJBzfRPZhZ5xcsKaSA5q4hPnzfK0x4ffzXx8+l2oazMCluY1M3n3zwdZbOL+GuG8/O+Gaj6RbNK+Y/P3oOR/tG+cj3N9KVQYkhu/6SKeiV/d28/56XGZv08fCnL+Dk2rJkh2TMjDZsbeVj979GXUUhD336fCqKM7O3UTjnLK3i/o+fy8GeIf74npfZ154ZTUmWFJJkbNLLt57Zw5/8cCPzivN49NYLWVNfnuywjAlpcGySLz2+nb98+A+sXVTJT/+/C6ktK0x2WEl10cnV/PhTFzA05uV93/s9D208hKomO6yoRDV0tpk9r095ansb33p2Dwe7h/njsxdxx3vXZGzfbpP+Jrw+Hn/jCN9+bg8dA2N8+u3L+cK7VpOfa78pAc5ZOo8n/+Ii/vpnW/m7x7fz6ObD/O21p3LusqpkhzYnlhQSpPXYCE9saeWR1w9xsHuYUxaU8eCfnsclq9J7EhKTuVp6h/nFH47w0MZDtPaNsm5xJXffdA7nLJ2X7NBSTl1FEf/9p+fz6OYWvvnsHj547yucvaSSG89bwjVn1FFakD5ftVFFKiIfBO4ATgXOU9VNIba7Gvg3wAP8QFX9M7RVAT8BlgHNwIdUtTeamFKBqtLWN8r2I328cbCXlxq72NnaD8D6pfP44tWrueq0hXish1HaClWmA9aLu/5aYBj4uKq+kfBAI6SqdAyMsa2lj03NPfxuXxe72pwy+7YV8/na9adz2Sm1NuTKDHJyhA+du5j3rK3jZ5taeODlZr7w6Da+9PgOzj+pigtXzOfsJfM4rb48pVsGok1fO4D3A/8ZagMR8QDfw5mOswV4XUQ2qOou4IvAC6p6p4h80X18e5QxxZSqMu71MTbpY3TCy8i4l6ExL0Pjk/SPTNA7PEHP0Bgd/WO09Y1yuHeYA51DDIw5Q+zmeYSzFs/j9qtXc/XpC1lebRekpbswZdrvGmClezsfuMf9PyF8Pqfcjk54GZnwMjQ2yeCY1y2z43QPjtMxMEZb3wiHeoZp6hyib2QCcMvsknn87TWrufaMOhZXFScq7IxQnJ/LzW9bxscuXMobh3p5Zmc7v93TwTee3jO1TX1FIcuqS1g0r4i6iiJqygqoKsmnsjiP8sI8SgpyKcn3UJjvoTDXQ55HEpaQo52OczcQLtjzgEZVbXK3fQS4Dtjl/n+pu92DwG+JIil85Puv0jEwdtyJHnX/USde939QFJ/PWeZT8Kri8yleVbw+ZdKrTPp8THgjO2lUmJdDfUURDfOKeN9ZDaxcUMpp9RWcVl+ecSNGmhnLtN91wI/UKYyvikiliNSpattsD7b5YC+3/3ybU34VfP4y61N8bnn1+pRJnzLp9THujazc5nmEhRWFLJ5XzLvPrGNVbSmnN1RwWn0FRflWZqMlIpyztIpzllbxd9eeSvegUxPb1dbPvvYBDvYM89s9nXQOjhHJuen83BzyPTnkeoTcHMGTI3hEyMkRcsR5LMB16xr47BVzv24kEQ1dDUDgIOQtvPWLaYH/Q6KqbSIScvQ3EbkFuAVgyZLgozAury5hXnG+u0PAvs7+7v8BjwVyRMgRZ9o9/xvryXHe9FxPDnmeHApy3Vueh6I8DyX5HkoLcykvzGNecT7zSvIoLci1qnX2mKlMz7RNA3BCUghXtovzPaxaUIrgfAEIkCNOc4UnoMzmeXLIzRHy3C+P/NwcCvM8FOc7tzK3zFYW5zPf/VVqZTZx5pcWcNnqWi6bNhT+pNdHz9A4PcPj9A5NMDA6weDYJMPjXkYnnNv4pI8xr8/5ser1MeFTvF7nR6xv6get8yO3riK6HmFhk4KIPA8sDLLqS6r6RATHCFbqZt1nS1XvA+4DWL9+fdD9//n6M2b7tMbMRSRlOuJyH65sn1pXzt03nTPbGE2ayPXkUFteSG15anTvDZsUVPWKKI/RAgQOnbgI8I8k1e6vUotIHdAR5bGMSYSZyvRstjEm5SSio/HrwEoRWS4i+cANwAZ33QbgZvf+zUAkNQ9jkm2mMu23AfiYOC4A+uZyPsGYRIsqKYjI9SLSAlwI/K+IPOMurxeRpwBUdRK4DXgG2A38VFX9g5HfCVwpIvtwenLcOf0YxqSaUGVaRG4VkVvdzZ4CmoBG4PvAnyclWGNmSdLxkuz169frpk1BL4kwJmoisllV1yfj2Fa2TTxFUrbtOnVjjDFTLCkYY4yZYknBGGPMFEsKxhhjpqTliWYR6QQOhlhdDXQlMJyZWCzBpXosS1U1KcPXzlC2U/09SxaLJbhQsYQt22mZFGYiIpuS1XNkOoslOItl9lIpTosluEyJxZqPjDHGTLGkYIwxZkomJoX7kh1AAIslOItl9lIpTosluIyIJePOKRhjjJm7TKwpGGOMmSNLCsYYY6akfVIQkTtE5IiIbHFv14bY7moR2SMije580PGI5Zsi8qaIbBORx0WkMsR2zSKy3Y03ZqOfhXuN7jDO/+6u3yYiZ8fq2NOOs1hEfiMiu0Vkp4h8Nsg2l4pIX8Df7R/jEYt7rBnf70S9L7Nh5fqE57ayHTye2JdtZ97X9L0BdwB/HWYbD7AfOAnIB7YCa+IQy1VArnv/68DXQ2zXDFTH+NhhXyNwLfArnFnBLgA2xulvUgec7d4vA/YGieVS4JcJKiMzvt+Jel9mGbOV61m8TivbIdfP+n1J+5pChKYmWlfVccA/0XpMqeqz6oy1D/AqzmxbiRLJa5yaTF5VXwUqxZnxLqZUtU1V33DvD+DMOdAQ6+PEUELelzjIhnINVrajMev3JVOSwm1u1eh+EZkXZH2oSdTj6U9xMnQwCjwrIpvFmbQ9FiJ5jQl/H0RkGXAWsDHI6gtFZKuI/EpETotjGOHe72SUj0hYuXZY2Q4t5mU77BzNqUBEngcWBln1JeAe4Ks4b85XgW/jFNzjniLIvnPqiztTLKr6hLvNl4BJ4MchnuYiVW0VkVrgORF5U1VfnEs8gaEFWTbnyeRjQURKgZ8Dn1PV/mmr38AZh2XQbS//BbAyTqGEe78T+r5MHdTKdcThBVlmZdsR87KdFklBVa+IZDsR+T7wyyCrYjaJerhYRORm4D3A5eo26gV5jlb3/w4ReRynehzthyelJpMXkTycD82PVfWx6esDP0iq+pSI3C0i1aoa8wHFIni/E/a+TIvLynVkrGyHEI+ynfbNR9Pax64HdgTZLJKJ1mMRy9XA7cB7VXU4xDYlIlLmv49zEi9YzLOVMpPJi4gAPwR2q+q/hthmobsdInIeTlnsjkMskbzfCXlfZsPK9XGsbAc/TnzKdiLOkMfzBvw3sB3Y5r4Bde7yeuCpgO2uxekpsB+nShyPWBpx2u+2uLd7p8eC04Niq3vbGctYgr1G4FbgVve+AN9z128H1sfpfbgYp4q6LeC9uHZaLLe5r38rzsnLt8UplqDvdzLeFyvXVrbToWzbMBfGGGOmpH3zkTHGmNixpGCMMWaKJQVjjDFTLCkYY4yZYknBGGPMFEsKxhhjplhSMMYYM+X/ByYUR2TUHXNWAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tanh函数图像\n",
    "plt.subplot(121)\n",
    "plt.plot(X.detach(), tanh_y.detach())\n",
    "plt.title(\"tanh Function\")\n",
    "# tanh导函数图像\n",
    "plt.subplot(122)\n",
    "plt.plot(X.detach(), X.grad.detach())\n",
    "plt.title(\"tanh Derivative function\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于tanh函数来说，导函数的取值分布在0-1之间的，看似导函数取值累乘之后也是趋于0的，但实际上，tanh激活函数的叠加即可能造成梯度消失、同时也可能造成梯度爆炸，原因是在实际建模过程中，影响前几层梯度的其他变量大多数情况都大于1，因此对于一个导函数极大值可以取到1的激活函数来说，还是有可能出现梯度爆炸的情况的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 梯度爆炸和梯度消失中，所谓的前几层参数梯度过大或者过小也都是相对的概念，并没有明确定义梯度大过多少就是梯度爆炸、梯度小过多少就是梯度消失。另外，梯度爆炸和梯度消失的直接表现可归结为前后参数层梯度不一致，而二者的根本问题都是影响迭代收敛过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 作为Sigmoid激活函数的“升级版”，tanh激活函数除了能够一定程度规避梯度消失问题外，还能够生成Zero-Centered Data，而确保输入层接收到Zero-Centered Data，则是解决梯度消失和梯度爆炸问题的关键。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "和Sigmoid激活函数一样，我们可以通过绘制小提琴图观察多轮迭代后，tanh激活函数叠加的神经网络是否会出现梯度消失或者梯度爆炸的情况。类似的，首先我们利用此前定义的模型，实例化一个拥有四个隐藏层的以tanh为激活函数的模型，然后观察模型在迭代过程中梯度变化情况。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从根本上来说，tanh激活函数的迭代不平稳就是因为部分层的部分梯度存在极端值，当然，这种极端值也导致部分层无法有效学习、最终影响模型效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ReLU 函数\n",
    "\n",
    "尽管ReLU在缓解梯度消失方面表现良好，但它并不能完全解决梯度爆炸问题。梯度爆炸通常发生在‌深层网络中，当权重初始化过大时，梯度在反向传播过程中会累积并变得非常大，导致模型无法收敛。为了应对梯度爆炸，除了使用ReLU激活函数外，还可以采取其他措施，如‌批量归一化（Batch Normalization）、‌梯度剪裁（Gradient Clipping）以及合理设置学习率等策略。这些方法共同作用，可以在一定程度上减轻梯度爆炸的影响，确保网络训练的稳定性和有效性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、Zero-Centered Data与Glorot条件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;通过对Sigmoid和tanh激活函数叠加后的模型梯度变化情况分析，我们不难发现，梯度不平稳是影响模型建模效果的非常核心的因素。而这个看似简单问题的解决方案，却花费了研究人员数十年的时间才逐渐完善，我们现在所接触到的优化方法，也基本上是在15年前后提出的，而这些被验证的切实可行的优化方法，也是推动这一轮深度学习浪潮的技术因素。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 当然，这些优化方法主要是针对深层次神经网络的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;整体来看，针对梯度不平稳的解决方案（优化方法）总共分为五类，分别是参数初始化方法、输入数据的归一化方法、衍生激活函数使用方法、学习率调度方法以及梯度下降优化方法。接下来，先介绍所有上述优化算法的一个基本理论，由Xavier Glorot在2010提出的Glorot条件。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 值得注意的是，虽然不同优化算法有不同的出发点和不同的论证方式，但基本都可以从Glorot条件出发进行思考。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.Zero-centered Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在介绍Glorot条件之前，我们先从一个更加朴素的角度出发，讨论关于Zero-Centered Data相关作用，从而帮助我们理解后续Glorot条件。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;首先，我们还是假设当前模型是一个三层神经网络，其中两个隐藏层的激活函数为$F(x)$，对应的导函数为$f(x)$，设X为输入训练的数据特征，y为标签，$\\hat y$为模型向前传播输出结果，$w_1$为第一层参数、$w_2$为第二层参数、$w_3$为第三层参数，loss为损失函数，则有如下计算公式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每一次正向传播计算结果："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\hat y = F(F(X * w_1) * w_2) * w_3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设$Z_i$为第i层接收到的数据，$P_i$为第i层输出的数据，则有："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Z_1 = X*w_1$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P_1 = F(Z_1) = F(X*w_1)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Z_2 = P_1 * w_2 = F(X*w_1)*w_2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P_2 = F(Z_2) = F(F(X*w_1)*w_2)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Z_3 = \\hat y = F(F(X * w_1) * w_2) * w_3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "依次类推。而在反向传播过程，各参数层的梯度如下"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "grad_1 \n",
    "= \\frac{\\partial loss}{\\partial \\hat y} \n",
    "\\cdot \n",
    "w_3 \\cdot f(F(X*w_1)*w_2) \\cdot\n",
    "w_2 \\cdot f(X * w_1) \\cdot X \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "grad_2 \n",
    "= \\frac{\\partial loss}{\\partial \\hat y} \n",
    "\\cdot \n",
    "w_3 \\cdot f(F(X*w_1)*w_2) \\cdot\n",
    "F(X * w_1) \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "grad_3 = \\frac{\\partial loss}{\\partial \\hat y} \n",
    "\\cdot \n",
    "F(F(X * w_1) * w_2)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在梯度消失和梯度爆炸的案例中，我们不难发现，为了确保多层神经网络的有效性，各层梯度的差距不应太大，此时一个最为基本的想法就是，就是能否让所有的输入数据（也就是X）以及所有层的参数都设置为Zero-Centered Data，也就是零点对称数据，不难发现，由于X和$w_i$都是零点对称的，因此每一个线性层中的导函数也取值也能够维持在0-1之间，进而每一层的梯度基本能维持在比较平稳的状态。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 另外，除了能够避免梯度不平稳问题以外，创建Zero-Centered的参数和数据集，还能够更好的在正向传播中将信息传播到各层，以及确保各层学习的平稳性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;关于如何将带入模型训练的数据转化为Zero-Centered Data，一般来说我们会使用一系列标准化方法对其进行转化，具体方法我们会在Lesson 14进行详细介绍，由于我们此前创建的数据生成器生成的就是Zero-Centered Data，因此暂时这些数据不会影响接下来的优化方法使用。而如何将参数转化为Zero-Centered Data，就是核心需要考虑的问题了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 对于输入的数据来说，我们可以尽量保证其Zero-Centered的特性，但模型参数是随着模型迭代不断变化的，我们无法把控模型每一轮迭代后的情况，因此只能从模型参数初始值入手，尽量保证其Zero-Centered属性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;当然，既然是创建Zero-Centered参数，那能否创建初始值全是0的参数呢？我们可以简单尝试一下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x26e5b4c16b0>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Sigmoid_class1_test(nn.Module):                                   \n",
    "    def __init__(self, in_features=2, n_hidden=2, out_features=1, bias=False):       \n",
    "        super(Sigmoid_class1_test, self).__init__()\n",
    "        self.linear1 = nn.Linear(in_features, n_hidden, bias=bias)\n",
    "        self.linear2 = nn.Linear(n_hidden, out_features, bias=bias)\n",
    "        \n",
    "    def forward(self, x):                                   \n",
    "        z1 = self.linear1(x)\n",
    "        p1 = torch.sigmoid(z1)                   \n",
    "        out = self.linear2(p1)\n",
    "        return out\n",
    "\n",
    "# 创建随机数种子\n",
    "torch.manual_seed(420) \n",
    "\n",
    "# 创建模型\n",
    "sigmoid_test = Sigmoid_class1_test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[ 0.4318, -0.4256],\n",
       "         [ 0.6730, -0.5617]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[-0.2157, -0.4873]], requires_grad=True)]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 观察各层参数\n",
    "list(sigmoid_test.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将各层参数修改为0\n",
    "list(sigmoid_test.parameters())[0].data = torch.tensor([[0., 0], [0, 0]])\n",
    "list(sigmoid_test.parameters())[1].data = torch.tensor([[0., 0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[0., 0.],\n",
       "         [0., 0.]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[0., 0.]], requires_grad=True)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看修改结果\n",
    "list(sigmoid_test.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 或者也可以通过torch.full_like(list(sigmoid_test.parameters())[0], 0)进行修改，然后再令其可导并替换原始参数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 遍历五次查看结果\n",
    "fit(net = sigmoid_test, \n",
    "    criterion = nn.MSELoss(), \n",
    "    optimizer = optim.SGD(sigmoid_test.parameters(), lr = 0.03), \n",
    "    batchdata = train_loader, \n",
    "    epochs=5, \n",
    "    cla=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[ 0.5914, -0.0281],\n",
       "         [ 0.5914, -0.0281]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[1.0375, 1.0375]], requires_grad=True)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(sigmoid_test.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们发现，参数的每一列（最后一个参数的一行）都是同步变化的，大家可以思考一下出现这种情况的原因是什么~？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;很明显，我们不能将参数的初始值全部设为0，我们只能考虑借助统计工具生成均值是0的随机数，也就是0均值的均匀分布或者是0均值的高斯分布，但这里需要考虑的另一个问题就是，该随机数的方差应该如何确定？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.Glorot条件和Xavier方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;初始化参数的方差如何确定这一问题在一个严谨论述如何保证模型有效性的论文中，从另一个角度出发，得到了回答。根据Xavier Glorot在2010年发表的[《Understanding the difficulty of training deep feedforward neural networks》](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)论文中的观点，为保证模型本身的有效性和稳定性，我们希望正向传播时，每个线性层输入数据的方差等于输出数据的方差，同时我们也希望反向传播时，数据流经某层之前和流经某层之后该层的梯度也具有相同的方差，虽然二者很难同时满足（除非相邻两层神经元个数相同），但Glorot和Bengio（论文第二作者）表示，如果我们适当修改计算过程、是可以找到一种折中方案去设计初始参数取值，从而同时保证二者条件尽可能得到满足，这种设计参数初始值的方法也被称为Xavier方法，而这种方法也经过一段时间的实践验证被证明是很好的一种初始化模型参数的方法，尤其是对于使用tanh激活函数的神经网络来说，效果更为显著。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;而这种正向传播时数据方差保持一致、反向传播时参数梯度方差保持一致的条件，也被称为Glorot条件，满足该条件的模型能够进行有效平稳的训练，而为了满足该条件而创建的（当然也是由上述论文提出的）模型初始化参数值设计方法，也被称为Xavier方法。而在Xavier方法中，最核心解决的问题，也就是为了创建Zero-Centered的初始化参数时参数的方差。和我们从朴素的角度思考的方向是一致的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;由于Glorot条件和Xavier方法是在2010年提出的，彼时ReLU激活函数还未兴起，因此Xavier方法主要是围绕tanh激活函数可能存在的梯度爆炸或梯度消失进行的优化，Sigmoid激活函数效果次之。不过尽管如此，Glorot条件却是一个通用条件，后续围绕ReLU激活函数、用于解决神经元活性失效的优化方法（如HE初始化方法），也是遵照Glorot条件进行的方法设计。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.模型初始化参数取值影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;Xavier初始化方法的推导和使用我们将在下一节详细介绍，此处我们先通过另外一个实例，去展示为何初始参数取值不同，会够得到不同的建模结果。模型初始化时得到的不同参数，本质上等价于在损失函数上找到了不同的初始点，而同一损失函数，初始点选取的不同应该不会影响最终迭代结果才对，但事实情况并非如此。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.loli.net/2021/01/23/caNAtMFI5dqhfH6.jpg\" alt=\"11\" style=\"zoom:40%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们通过一个实验来说明初始值的更换对模型结果的影响。在模型实例化过程中，采用不同随机数种子，就相当于选取了不同的模型初始参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x26e5b4c16b0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x26e621eaa88>]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x26e62bb8688>]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x26e18a2f048>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAngElEQVR4nO3deXzV9Z3v8dcnO1mAJCSALIYlIgKCGre6LyAug8vj2mmddhynLZ07znWZa1Vura0dbe303tZxrsvlWm1HHWtLdcaO1qLW1o4XF2JBQFCQRUIiCSFwEhLI9r1/nJMYwtlyzu9snPfz8cjjJL/zO+f3zY/j22++v8/v+zXnHCIiknlyUt0AERGJjQJcRCRDKcBFRDKUAlxEJEMpwEVEMlReMg82btw4V1NTk8xDiohkvPr6+j3Ouarh25Ma4DU1NaxevTqZhxQRyXhmtiPYdg2hiIhkKAW4iEiGUoCLiGSoiAFuZo+bWbOZrR+y7R/M7H0zW2NmK83smMQ2U0REhoumB/5TYPGwbT90zp3onFsA/Adwt8ftEhGRCCIGuHPuDWDvsG2+IT+WAJoRS0QkyWIuIzSz+4C/BPYDF4TZbymwFGDq1KmxHk5ERIaxaKaTNbMa4D+cc3ODPLcMKHLOfTvS+9TV1bmY6sD3boX+PhhXO/LXHg32fQJ/ehpcf6pbIiKxmv8FqJwR00vNrN45Vzd8uxc38vwr8CIQMcBj9od/hLXPwIwL4bSvQ+0iyMmiApp3fwJvPgBYqlsiIrGacnrMAR5KTAFuZrXOuc2BH5cAm7xrUhALvwsV02H14/DMn0N5DZz6NTjpSzBqbEIPnRZ8u/y/881rU90SEUkj0ZQRPgOsAmaZWYOZfQW438zWm9n7wCLg5oS2srQazrsdblkH/+UJKJsIK78JP5oNv74Fdn+Q0MOnnK8JylSpKSKHi9gDd859McjmnySgLZHl5sPca/xfTWvhneX+oZX6J6DmHDhtKcy6DHKTOsVL4vl2waSTU90KEUkzmTuQPHE+XPkQ/P1GuPg70LYdfvFleHAB/PFHcKA1xQ30iHPQ3uT/q0NEZIjMDfABxRVw9q3+8eE/fxoqpsFr9/iHV/7tRn9PPZN1tUHvQRitIRQROdzRM9aQkwuzr/B/NW8MDK/8HNY85b/6e/4ymBGyXD19tTf5HxXgIjJM5vfAg6meDVf82D+8csn3wNcIz34Z+jOwjtoXCHBdxBSRYY7OAB8waiyceSOcfQt0t0PHp6lu0ci1N/ofR2sMXEQOd3QH+ICK6f7HvVtT245YDPTASyekth0iknYU4OnOtwtKqiCvINUtEZE0kx0BPmYK5ORnZoCrhFBEQsiOAM/J9d+KnokB7mtSBYqIBJUdAQ7+YZRMDPD2RgW4iASVZQG+zX9nY6boPQSdrSohFJGgsivAuzugoznVLYne4E08GgMXkSNlV4BDZg2jDN7EowAXkSNlUYBP8z9mVIDv8j9qDFxEgsieAB87FSw3swK8XT1wEQktewI8N98f4pkU4L4myC+GojGpbomIpKHsCXDIvFLCgRJC01qYInKkLAzwDCol9OkuTBEJLfsC/NB+6Nyb6pZERzfxiEgY2RfgkBnDKP396oGLSFjRrEr/uJk1m9n6Idt+aGabzOx9M3vezMYmtJVeyaQA72yF/h71wEUkpGh64D8FFg/b9gow1zl3IvARsMzjdiVG+bFgOZkR4AMLOagHLiIhRAxw59wbwN5h21Y653oDP74FTE5A27yXVwhjJmdGgPu0FqaIhOfFGPhfA78J9aSZLTWz1Wa2uqWlxYPDxSlTSgkHl1JTgItIcHEFuJl9E+gFng61j3NuuXOuzjlXV1VVFc/hvJEpAe5r8g/3lFSnuiUikqZiDnAzux64AvgL5zKlsBp/gHftha62VLckvPZGKB0PuXmpbomIpKmYAtzMFgN3AEucc53eNinBMqUSxdeoC5giElY0ZYTPAKuAWWbWYGZfAf43UAa8YmZrzOzRBLfTO4MBvi217YhES6mJSAQR/z53zn0xyOafJKAtyVFe439M9x54eyPUnJ3qVohIGsuuOzEB8kfB6EnpHeDdnXBwv1biEZGwsi/AIf0rUQaXUpuU2naISFrL0gCflt4B7tNdmCISWZYG+HQ40AIHfaluSXA+3cQjIpFlb4ADtKVpJYrmQRGRKGRpgM/wP6brMIqvCQpHQ2FpqlsiImksSwM8zVeob9dNPCISWXYGeEEJlE5I3wDXTTwiEoXsDHD4bH3MdNSuABeRyLI8wNOwB97fB+2faghFRCLK4gCf5u/pdh9IdUsO19EMrk93YYpIRFkc4Gk6qdVgCaGGUEQkPAV4ug2jDC6lph64iISXxQGepqWEA/OgqAcuIhFkb4AXjYHicekX4L5GyMmDkjRYfk5E0lr2BjikZyVKe5O/AiUnu/9pRCSy7E6JdKwF9+1SCaGIREUB7muAnq5Ut+QzviZdwBSRqCjAAdp2pLYdQ7U36QKmiEQluwO8Ms1KCQ/6oLtDPXARiUp2B3i61YKrhFBERiBigJvZ42bWbGbrh2y71sw2mFm/mdUltokJNKrc/5UuAa6VeERkBKLpgf8UWDxs23rgGuANrxuUdOlUStiuuzBFJHoRA9w59wawd9i2jc65DxPWqmRKpwD37fI/qoxQRKKQ8DFwM1tqZqvNbHVLS0uiDzdyFdNh/07o7U51S/wlhKPKIX9UqlsiIhkg4QHunFvunKtzztVVVaXh7eEV08H1w740KCVUCaGIjEB2V6FAelWi+Bo1/i0iUVOAp1OAD8yDIiIShWjKCJ8BVgGzzKzBzL5iZlebWQNwJvCimf020Q1NmOJKKByd+gDv6/GvxjN6UmrbISIZIy/SDs65L4Z46nmP25IaZv65wVMd4B27AachFBGJmoZQID1KCX1aSk1ERkYBDv4A3/eJfxgjVQbvwlQPXESiowAHqJgB/b3+evBU0TwoIjJCCnBIj0oUXyPkFkJxReraICIZRQEOQwI8havztDdB2QT/RVURkSgowAFKqyG/JMU98CaVEIrIiEQsI0wHqz5uZde+Ls6fVcW40kLvD2CW+koU3y6YdHLQp3a0HuBf3/kE55LcJhHxzOfrpjCzutTT98yIAH/+Tw38YnUDZjB/8lguPL6aC4+vZs4xozGvhhwqpkHLJm/ea6ScC3sX5pOrdvDYf25jVH5ukhsmIl45p3Zcdgb4/decyJfPqOF3m5r53YfN/PjVj/jRKx8xfnQhF8yq5oLjqzl75jhKCuP4dSqmw0cvQ38f5CQ5KLvaoPdgyIUcGtq6mFldyqt/f15y2yUiaS0jAjwnx5g3eQzzJo/h5otraWk/xO8/bOb1D5t58f0mfv7uTgpyczh9esVg7/zYypKRHaRiOvR1w/4GKD82Mb9IKIMlhMF74A37OplcrilmReRwGRHgw1WVFXJt3RSurZtCT18/727fy+ubmvndpmbu+fUH3PPrD5heVcKFs/xhXldTQUFehOu1Q0sJkx3gvoGVeEL3wBdMGZu89ohIRsjIAB8qPzeHz80Yx+dmjOObl5/AjtYD/qGWTc38S2DsuKwwj2vrpnD3n50Q+o2GBviMC5LT+AHtA7fRH9kDbz/Yw77OHiaXFye3TSKS9jI+wIc7trKEG86axg1nTePAoV7e3LKHx9/cxhP/bxu3L55FUagLgWUTIa8oNZUovtBDKA1tXQAaQhGRIxzVdeAlhXksmjOBL59Rg3PwcUtH6J1zcqB8Wmpu5vHtgpIqyCs44qnPAlw9cBE53FEd4ANqx/tLd7Y0hwlwSF0teJgSwoa2TgCmqAcuIsNkRYDXVJaQm2Ns3h0hwCunQ9s26O9PTsMG+JrCXsAclZ9LRcmRvXMRyW5ZEeAFeTnUVBazubk9/I4V0/312ANlfcnS3hi2Bz65fJR3NyyJyFEjKwIcoLa6jM3RDKFAcodReg9BZ2vYHrguYIpIMNkT4ONL2dHayaHevtA7pSLAI93E09alC5giElTWBPjM6lL6+h3b93SG3mn0JMgtSG6AD67Ec2QP3Hewh/1dPeqBi0hQ0axK/7iZNZvZ+iHbKszsFTPbHHgsT2wz41dbXQYQfhw8JxfKa9ImwHephFBEwoimB/5TYPGwbXcCrznnaoHXAj+ntelVJeQYkStRKqYntxY8zBCKbuIRkXAiBrhz7g1g77DNVwI/C3z/M+Aqb5vlvaL8XKZWFEdfC56sybd9TZBfDEVjjnhq517/cI8CXESCiXUMfLxzrgkg8FgdakczW2pmq81sdUtLS4yH88bM6rLoSgl7DkDH7uQ0aqCEMEiZYENbF8UFqgEXkeASfhHTObfcOVfnnKurqqpK9OHCqh1fyrY9B+jpC3OjTsU0/2OyxsHD3sSjGnARCS3WAN9tZhMBAo/N3jUpcWqrS+npc+xoDVOJkuxSwrA38aiEUERCizXAXwCuD3x/PfDv3jQnsQYqUbaEG0YZMxVy8pIT4P39UfXARUSCiaaM8BlgFTDLzBrM7CvA/cBCM9sMLAz8nPZmVPtX6QlbiZKbB2OnJifAO1uhvydogO/v6sF3sFcBLiIhRZwP3Dn3xRBPXeRxWxKuuCCPyeWjorulPhkBHmYhB9WAi0gkWXMn5oDa6tIoAnyGvxY80aWEYZZSG5hGVj1wEQkl+wJ8fBkft3TQ1x8mnCumwyGff4gjkcL0wLWQg4hEknUBPrO6lO7e/sGbZIJKViWKrwksB0rHH/HUzrZOigtyKS/OT2wbRCRjZV2A11b7V+cJO4yStABvhJJq/4XTYRrauphSXqwacBEJKesCfOZggIcpJRw71d8zTnSAtzdqHnARiVnWBXhZUT4TxxSxJVwpYV4BjJmSnCEU1YCLSIyyLsDB3wtPi1LCEHdh7u/qof1gry5gikhYWRngtdVlbGnuoD9SJUoiA7y7Ew7uh9HBKlBUQigikWVngI8vpaunj137ukLvVDEdutqgc/hMuh4ZnAc8WA24SghFJLLsDPDAhcywc4MPVqIkaHGHwZV4tJCDiMQmKwM8qkqURJcSDgR40B54JyUFuYxVDbiIhJGVAT62uICqssLwk1qV1wCWuABvD90D37m3iykVqgEXkfCyMsAhijlR8ov8q9QnrAfeBIWjobDsiKdUQigi0cjqAN/S3IELN2FVZQIrUUKUEDrn2KWFHEQkClkb4DPHl9FxqJdPfQdD75TIUkJfU9DhE19XL+2HNA+4iESWtQE+OCdKuHHwiunQucdfr+219qagFzB3qgZcRKKkAE9FKWF/H7R/GqGEUEMoIhJe1gZ4ZWkhFSUF4dfHTFQpYUczuL4Q84CrBy4i0cnaAIfAnCgRSwnxPsAHSwgnHfFUQ1sXpYV5jBmlGnARCS+rA3yglDBkJUpBib+X7PUQyuBSasGHUCaXj1INuIhElPUBvr+rh5aOQ6F3SkQlSth5UDo1/i0iUYkrwM3sZjNbb2YbzOwWj9qUNLXj/TfRhJ0bvGKa9wHua4ScPCipOmyzc04LOYhI1GIOcDObC3wNOA2YD1xhZrVeNSwZoq5E6fgUDkWYP3wk2pugdALkHH7693f10KEacBGJUjw98NnAW865TudcL/AH4GpvmpUcVWWFjC7Ki25SqzYPx8F9u1RCKCJxiyfA1wPnmlmlmRUDlwFThu9kZkvNbLWZrW5paYnjcN4zM2rHl0W+mQe8HUYJsZSaSghFZCRiDnDn3EbgB8ArwMvAWqA3yH7LnXN1zrm6qqqq4U+n3MCcKCGVT/M/ehngIe7CHOiBT1EPXESiENdFTOfcT5xzJzvnzgX2Apu9aVbyzKwupfVAN62hKlGKRvsvNnoV4Ad90N0RcgilrDCP0aPyvDmWiBzV4q1CqQ48TgWuAZ7xolHJNFiJEvZC5gzvasEjlBBOUg24iEQp3jrwX5nZB8CvgRudc20etCmpoq5E2bMZwk09G60IS6npAqaIRCveIZRznHMnOOfmO+de86pRyTRxTBElBbnhe+A1Z/lLCT/8TfwHHOyBHx7gAzXgUyp0AVNEopPVd2KCvxJl5viy8KWEJ37BP4zyu3/wzyQYD98u/+OwKpR9nQM14OqBi0h0sj7AITAnSrhSwtw8uPAuaP4A1v0yvoP5mmBUOeQf3tPWSvQiMlIKcPwB3tx+iP2dPaF3OuEqmHAivH4f9HbHfrCQJYSqAReRkVGAA7Xj/Rcyt7SEGUbJyYGLvw37PoH6n8Z+MF+j7sIUEU8owIHaan8pYdhhFIAZF8GxZ8MbP4x9bpT2ppALOZQVaR5wEYmeAhyYNHYURfk54UsJAcz8vfADzfD2IyM/UF+PfzWeoLfRq4RQREZGAQ7k5Jh/dZ5IAQ4w5TSYdRm8+SB07h3Zgdo/BVyIHrimkRWRkVGAB9RWl7Fld5gx8KEu/BYcaof//PHIDjJQAz6sB+6vAe9UgIvIiCjAA2ZWl9K4/yDtB8NUogwYfwKc+OfwzvLP7qyMxuBdmEfWgB/o7tMkViIyIgrwgIFb6j9uORDdCy5Y5r+p5w8/iP4gIeZB2akSQhGJgQI8YGBSq83RDqOU10DdDfDek9D6cXSv8TVCbiEUVxy2WSWEIhILBXjAlPJRFOTlhJ8TZbhzvwF5RfC7e6Pbv70Jyib4q1mGGLiJZ5J64CIyAgrwgLzcHKaPK4muEmVAaTWc8V9hw3PQtDby/iFX4ulitGrARWSEFOBD1Eaa1CqYs27yz23y2ncj7+vbFaaEUMMnIjIyCvAhaqtLaWjrorP7iJXhQisaA2ffCltehe3/GXo/5/xDKCHWwtQFTBEZKQX4ELXVpTgHW6OtRBlw2lJ/z/rVe0Iv+tDVBr0HQ9SAqwcuIiOnAB9iYFKrEQ+j5I+C8+6Ahnfgo5eD7xNiIYe2zh46u/vUAxeREVOAD3FsZQl5ORZ5UqtgTvqSf9GH174bfNEHX/C7MAcqUKZUqAcuIiOjAB8iPzeHaSOtRBmQmz9k0YcVRz7fHrgLc1gPXAs5iEisFODD1I4vHVkt+FDhFn3wBQ/wnXtVAy4isYkrwM3sVjPbYGbrzewZMyvyqmGpMrO6jB2tBzjYE8Pal4OLPuyA9352+HO+RigeB3kFh21uaOtizKh8RhepBlxERibmADezScBNQJ1zbi6QC3zBq4alSm11Kf0Otu0ZYSXKgIFFH/7wj9A95D3am0KsxKMSQhGJTbxDKHnAKDPLA4qBEUzNl54+q0SJcRhl6KIPbw1Z9MHXBKMnHbG75gEXkVjFHODOuV3A/wQ+AZqA/c65lcP3M7OlZrbazFa3tLTE3tIkmTauhBwj+rnBgwm26EN74xHj36oBF5F4xDOEUg5cCUwDjgFKzOxLw/dzzi13ztU55+qqqqpib2mSFOblUlMZYyXKUBd+Cw754M0HoPcQdLYeUUK490A3XT2qAReR2MQzhHIxsM051+Kc6wGeAz7nTbNSa0a0y6uFM7Dow9v/B3a9598WsoRQPXARGbl4AvwT4AwzKzYzAy4CNnrTrNSqrS5l+54DdPf2x/dGA4s+vPQN/8+jgwf4lAr1wEVk5OIZA38bWAG8B6wLvNdyj9qVUrXjS+ntd+xojbESZcDAog+71/l/Lgt+F+aksQpwERm5uKpQnHPfds4d75yb65z7snPukFcNS6Xa6sDqPPEOo4B/0Yf8Ev/3w3rgO9s6GVucT5lqwEUkBroTM4gZVaWYEducKMOVVsN534DKWigae9hTKiEUkXgowIMYVZDL5PJRI5+VMJSzb4W/ezfIUmpdTB6rC5giEhsFeAi11WWxz4kSzLDw9teA6y5MEYmdAjyE2upStrYcoLcvzkqUEFoPdHOwp18BLiIxU4CHMLO6lO6+fj4JzBboNdWAi0i8FOAh1I73sBIliIESwsmqAReRGCnAQ5hZ7Z/UytNx8CHUAxeReCnAQygtzOOYMUVsjmdSqzAa2jopL86ntDAvIe8vIkc/BXgYM8eXJWwIZedezUIoIvFRgIdRW+1fXq2v33n+3iohFJF4KcDDqK0u5VBvP7sC49Ve+WwecAW4iMROAR7GZ6vzeDsOvqejm0O9/RpCEZG4KMDDmFmVmFLCwRJC9cBFJA4K8DDGFOdTXVbozaRWQ6iEUES8oACPoHZ8KVs8HkL5LMDVAxeR2CnAI6it9pcSOuddJUpDWycVJQWUqAZcROKgAI9gZnUpnd19NO4/6Nl7qgJFRLygLmAEtYFb6jfvbvds6bOdbZ0cP6HMk/cSSZWenh4aGho4eNC7zk22KyoqYvLkyeTnR7dKlwI8goFJrbY0d3D+rOq43885x662Li6ePT7u9xJJpYaGBsrKyqipqcGGzXcvI+eco7W1lYaGBqZNmxbVazSEEkFFSQGVJQWeVaK0dBwK1IBrCEUy28GDB6msrFR4e8TMqKysHNFfNArwKMysLvXsZh5VoMjRROHtrZGez5gD3MxmmdmaIV8+M7sl1vdLZ8eNL2PTp+20HeiO+71UAy4iXok5wJ1zHzrnFjjnFgCnAJ3A8141LJ38xRlT6e7t576XNsb9XgN3YXp1QVREspdXQygXAR8753Z49H5p5fgJo/n6edNZUd/Am1v2xPVeDW1dVKoGXMQT+/bt4+GHHx7x6y677DL27dvnfYOSzKsU+QLwTLAnzGwpsBRg6tSpHh0u+f7bhbW8+H4T/+P5dfz2lnMpys+N6X1UAy5Ho3t+vYEPGn2evucJx4zm2382J+w+AwH+t3/7t4dt7+vrIzc39H+jL730kidtTLW4e+BmVgAsAX4Z7Hnn3HLnXJ1zrq6qqirew6VMUX4u37t6HjtaO/mn1zbH/D7+ecA1/i3ihTvvvJOPP/6YBQsWcOqpp3LBBRdw3XXXMW/ePACuuuoqTjnlFObMmcPy5csHX1dTU8OePXvYvn07s2fP5mtf+xpz5sxh0aJFdHWFnj76/PPP59Zbb+Xcc89l9uzZvPvuu1xzzTXU1tZy1113AXDgwAEuv/xy5s+fz9y5c3n22WcBqK+v57zzzuOUU07hkksuoampKf4T4JyL6wu4ElgZzb6nnHKKy3S3/WKNm77sRfdB4/4Rv7a/v98d982X3Pde/CABLRNJrg8+SP3neNu2bW7OnDnOOedef/11V1xc7LZu3Tr4fGtrq3POuc7OTjdnzhy3Z88e55xzxx57rGtpaXHbtm1zubm57k9/+pNzzrlrr73WPfnkkyGPd95557nbb7/dOefcAw884CZOnOgaGxvdwYMH3aRJk9yePXvcihUr3Fe/+tXB1+zbt891d3e7M8880zU3NzvnnPv5z3/ubrjhhqDHCHZegdUuSKZ6MQb+RUIMnxyN/sdlsxk7Kp87n1s34pV6WtpVAy6SSKeddtphN8E8+OCDzJ8/nzPOOIOdO3eyefORfz1PmzaNBQsWAHDKKaewffv2sMdYsmQJAPPmzWPOnDlMnDiRwsJCpk+fzs6dO5k3bx6vvvoqd9xxB3/84x8ZM2YMH374IevXr2fhwoUsWLCAe++9l4aGhrh/37gC3MyKgYXAc3G3JEOUlxRw95+dwNqd+/iXVdtH9NqdKiEUSaiSkpLB73//+9/z6quvsmrVKtauXctJJ50U9CaZwsLCwe9zc3Pp7e0Ne4yB/XNycg57bU5ODr29vRx33HHU19czb948li1bxne/+12cc8yZM4c1a9awZs0a1q1bx8qVK+P9deMLcOdcp3Ou0jm3P+6WZJAl84/hvOOq+OFvP2TXvuiXW9NCDiLeKisro709+E12+/fvp7y8nOLiYjZt2sRbb72VlDY1NjZSXFzMl770JW677Tbee+89Zs2aRUtLC6tWrQL888hs2LAh7mOpli0GZsa9V81l0Y/f4O5/W89j19dFdQfVwE08kxTgIp6orKzkrLPOYu7cuYwaNYrx4z+bY2jx4sU8+uijnHjiicyaNYszzjgjKW1at24d3/jGN8jJySE/P59HHnmEgoICVqxYwU033cT+/fvp7e3llltuYc6c8FU2kZjzcJ7rSOrq6tzq1auTdrxEe+yPW7n3xY08dN3JXH7ixIj7L3tuHSs3fEr9txYmoXUiibVx40Zmz56d6mYcdYKdVzOrd87VDd9Xc6HE4a8+V8O8SWP49gsb2N/ZE3H/hrZOJldo/FtEvKEAj0Nebg7fv2YebZ3d3P9y5Nvsd+kmHpGMcOONN7JgwYLDvp544olUN+sIGgOP09xJY/jK2dNY/sZWrlowidOnVwbdr7/f0bCvi4VzNA+4SLp76KGHUt2EqKgH7oFbLq5lSsUolj2/joM9fUH32dNxiO7efpUQiohnFOAeKC7I476r5rG15QAP//7joPvsVAmhiHhMAe6Rc4+r4uqTJvHI77fw0e4j61IHSginKMBFxCMKcA/ddflsSgvzWPbcOvqH3WY/WAM+VkMoIl6JdTpZgAceeIDOzk6PW5RcCnAPVZYWctflJ1C/o41/feeTw55raOtkXGkBowpim4ZWRI6U7QGuKhSPXXPyJJ77UwM/+M0mLp49ngljigB/D3ySLmDK0eo3d8Kn67x9zwnz4NL7w+4ydDrZhQsXUl1dzS9+8QsOHTrE1VdfzT333MOBAwf4/Oc/T0NDA319fXzrW99i9+7dNDY2csEFFzBu3Dhef/31oO9fWlrKjTfeyKuvvkp5eTnf+973uP322/nkk0944IEHWLJkCRs2bOCGG26gu7ub/v5+fvWrX1FbW8tTTz3Fgw8+SHd3N6effjoPP/xw2DnKY6EeuMfMjPuumkd3Xz/feeGzuQ4a2ro0/i3isfvvv58ZM2awZs0aFi5cyObNm3nnnXdYs2YN9fX1vPHGG7z88sscc8wxrF27lvXr17N48WJuuukmjjnmGF5//fWQ4Q3+ub3PP/986uvrKSsr46677uKVV17h+eef5+677wbg0Ucf5eabb2bNmjWsXr2ayZMns3HjRp599lnefPNN1qxZQ25uLk8//bTnv7964AlQM66EWy4+jh+8vInfbviUhbPHs6uti0vmTEh100QSI0JPORlWrlzJypUrOemkkwDo6Ohg8+bNnHPOOdx2223ccccdXHHFFZxzzjlRv2dBQQGLFy8G/NPHFhYWkp+fz7x58wannT3zzDO57777aGhoGFzc4bXXXqO+vp5TTz0VgK6uLqqrq739hVGAJ8xXz5nGC2sbufvf1zOjqpTuPs0DLpJIzjmWLVvG17/+9SOeq6+v56WXXmLZsmUsWrRosPccSX5+/uBEdUOnjx2YOhbguuuu4/TTT+fFF1/kkksu4bHHHsM5x/XXX8/3v/99j3674DSEkiD5uTncf808mtsP8d9/uRZQDbiI14ZOJ3vJJZfw+OOP09HRAcCuXbtobm4OOr3r8NfGY+vWrUyfPp2bbrqJJUuW8P7773PRRRexYsUKmpubAdi7dy87dni/5rt64Ak0f8pY/upzNTzx5nZACzmIeG3odLKXXnop1113HWeeeSbgvwD51FNPsWXLliOmdwVYunQpl156KRMnTgw7Dh7Js88+y1NPPUV+fj4TJkzg7rvvpqKignvvvZdFixbR399Pfn4+Dz30EMcee6wnv/cATSebYAcO9bLox2+wa18Xm/5hccyr2YukG00nmxgjmU5WPfAEKynM45+vO4lVH7cqvEXEUwrwJDh5ajknTy1PdTNEJITTTz+dQ4cOHbbtySefZN68eSlqUXQU4CKS9d5+++1UNyEm8a5KP9bMVpjZJjPbaGZnetUwEUl/ybyGlg1Gej7jLSP8J+Bl59zxwHwg8rI0InJUKCoqorW1VSHuEeccra2tFBUVRf2amIdQzGw0cC7wV4GDdwPdsb6fiGSWyZMn09DQQEtLS6qbctQoKipi8uTJUe8fzxj4dKAFeMLM5gP1wM3OuQNDdzKzpcBSgKlTp8ZxOBFJJ/n5+UybNi3Vzchq8Qyh5AEnA484504CDgB3Dt/JObfcOVfnnKurqqqK43AiIjJUPAHeADQ45wYu367AH+giIpIEMQe4c+5TYKeZzQpsugj4wJNWiYhIRHHdSm9mC4DHgAJgK3CDc64tzP4tQKwzuowD9sT42mRQ++Kj9sVH7YtfOrfxWOfcEWPQSZ0LJR5mtjrYXADpQu2Lj9oXH7UvfpnQxuE0nayISIZSgIuIZKhMCvDlqW5ABGpffNS++Kh98cuENh4mY8bARUTkcJnUAxcRkSEU4CIiGSrtAtzMFpvZh2a2xcyOuDXf/B4MPP++mSXt7k8zm2Jmrwemzt1gZjcH2ed8M9tvZmsCX9Etf+1dG7eb2brAsY9Yvy7F52/WkPOyxsx8ZnbLsH2Sev7M7HEzazaz9UO2VZjZK2a2OfAYdDWOSJ/VBLbvh4EpnN83s+fNbGyI14b9LCSwfd8xs11D/g0vC/HaVJ2/Z4e0bbuZrQnx2oSfv7g559LmC8gFPsY/UVYBsBY4Ydg+lwG/AQw4A3g7ie2bCJwc+L4M+ChI+84H/iOF53A7MC7M8yk7f0H+rT/Ff4NCys4f/hk1TwbWD9n2j8Cdge/vBH4Qov1hP6sJbN8iIC/w/Q+CtS+az0IC2/cd4LYo/v1Tcv6GPf+/gLtTdf7i/Uq3HvhpwBbn3Fbnn57258CVw/a5EvgX5/cWMNbMJiajcc65Jufce4Hv2/HPfz4pGcf2UMrO3zAXAR8752K9M9cTzrk3gL3DNl8J/Czw/c+Aq4K8NJrPakLa55xb6ZzrDfz4FhD9/KMeC3H+opGy8zfAzAz4PPCM18dNlnQL8EnAziE/N3BkQEazT8KZWQ1wEhBsLaYzzWytmf3GzOYkt2U4YKWZ1Qem8h0uLc4f8AVC/4eTyvMHMN451wT+/2kD1UH2SZfz+Nf4/6IKJtJnIZH+LjDE83iIIah0OH/nALudc5tDPJ/K8xeVdAtwC7JteJ1jNPsklJmVAr8CbnHO+YY9/R7+YYH5wD8D/5bMtgFnOedOBi4FbjSzc4c9nw7nrwBYAvwyyNOpPn/RSofz+E2gF3g6xC6RPguJ8ggwA1gANOEfphgu5ecP+CLhe9+pOn9RS7cAbwCmDPl5MtAYwz4JY2b5+MP7aefcc8Ofd875nHMdge9fAvLNbFyy2uecaww8NgPP4/9TdaiUnr+AS4H3nHO7hz+R6vMXsHtgWCnw2Bxkn1R/Dq8HrgD+wgUGbIeL4rOQEM653c65PudcP/B/Qxw31ecvD7gGeDbUPqk6fyORbgH+LlBrZtMCvbQvAC8M2+cF4C8D1RRnAPsH/txNtMCY2U+Ajc65H4XYZ0JgP8zsNPznuDVJ7Ssxs7KB7/Ff7Fo/bLeUnb8hQvZ8Unn+hngBuD7w/fXAvwfZJ5rPakKY2WLgDmCJc64zxD7RfBYS1b6h11SuDnHclJ2/gIuBTc65hmBPpvL8jUiqr6IO/8JfJfER/ivU3wxs+xvgbwLfG/BQ4Pl1QF0S23Y2/j/z3gfWBL4uG9a+vwM24L+q/hbwuSS2b3rguGsDbUir8xc4fjH+QB4zZFvKzh/+/5E0AT34e4VfASqB14DNgceKwL7HAC+F+6wmqX1b8I8fD3wGHx3evlCfhSS178nAZ+t9/KE8MZ3OX2D7Twc+c0P2Tfr5i/dLt9KLiGSodBtCERGRKCnARUQylAJcRCRDKcBFRDKUAlxEJEMpwEVEMpQCXEQkQ/1/YWh9o0viTBsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 创建随机数种子\n",
    "torch.manual_seed(420)  \n",
    "\n",
    "# 实例化模型\n",
    "relu_model3 = ReLU_class3(bias=False)              \n",
    "\n",
    "# 核心参数\n",
    "num_epochs = 20\n",
    "lr = 0.03\n",
    "\n",
    "# 模型训练\n",
    "train_l, test_l = model_train_test(relu_model3, \n",
    "                                   train_loader,\n",
    "                                   test_loader,\n",
    "                                   num_epochs = num_epochs, \n",
    "                                   criterion = nn.MSELoss(), \n",
    "                                   optimizer = optim.SGD, \n",
    "                                   lr = 0.03, \n",
    "                                   cla = False, \n",
    "                                   eva = mse_cal)\n",
    "\n",
    "# 绘制图像，查看MSE变化情况\n",
    "plt.plot(list(range(num_epochs)), train_l, label='train_mse')\n",
    "plt.plot(list(range(num_epochs)), test_l, label='test_mse')\n",
    "plt.legend(loc = 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x24432cad570>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x244657d0208>]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x24465785a08>]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x244657da888>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzWUlEQVR4nO3deXxU1dnA8d+ZyUwm+w4JJECAgBBiEMKmslhlU+tWrdZqKVWxrX1d+lqXVm3t3tr3fe3iUlu3glvdrYICisUqIAkGEjYDCCQQspKE7Muc9487CSFmGTJ75vl+Pvdz79x7594nN5MnZ8499xyltUYIIUTgMfk6ACGEEIMjCVwIIQKUJHAhhAhQksCFECJASQIXQogAFeLNkyUmJuoxY8Z485RCCBHw8vLyKrXWST3XezWBjxkzhtzcXG+eUgghAp5S6lBv66UKRQghApQkcCGECFCSwIUQIkBJAhdCiAAlCVwIIQKUJHAhhAhQksCFECJABUYCL1oHmx6FuqO+jkQIIfzGgAlcKfWUUqpcKVXYbd0vlFI7lFL5Sqm1SqkRHo3y8/fgvXvhfyfD0xfB1r9DQ6VHTymEEP5ODTSgg1JqHlAP/ENrPcWxLlprXedYvhWYrLX+7kAny8nJ0YN+ErOyCApfg8JXoXIvKDOkz4MpX4NJF0NY3OCOK4QQfk4plae1zum5fsASuNZ6I1DdY11dt5cRgOeH9UnMgAV3wy1b4HufwLm3w/GD8NYP4KEMeP4a2PFPaDnh8VCEEMIfDLovFKXUr4BvAbXAef3stwJYATBq1KjBnq77AWF4pjF95X44+plRKt/5Ony+BkJsMGExZF5hzC1hrp9TCCH80IBVKABKqTHA251VKD223QvYtNY/Heg4LlWhDMRuh5JPHcn8DWgoB2skTFxqVLOM+wqEhHrm3EII4UF9VaG4I4GPBt7pbVtPHk3g3dk74OB/jGS++y1oOg6WcEjNgbTZMGo2pM4AW7TnYxFCCBf1lcAHVYWilMrQWhc5Xl4C7HElOLczmWHsfGO66H/gwIdQtBYOb4aP/gDaDspkVMN0JvRRsyEm1deRCyGE0wZM4EqpF4AFQKJSqgT4KXChUmoiYAcOAQO2QPEZswUyFhoTGDc5S7bC4S1weBPkPw9b/2Zsi0mDtFknE/qwycY/AyGE8ENOVaG4i9eqUE5HRzuUFRgJvXizUUo/UWpsC402qlpGzYbECdDeAm0N0NoIbY3Q2uCYN/az3jHFpMKKD8Ea4dMfVwgReFyqA3cXv0zgPWkNNYeNRN6Z0Mt302tLSbPVqFu3Rjjm4WCJMFq+dC5bwwEFuU/C/LvhvB97+ycSQgQ4t9aBD2lKQdxoY8q+2ljXVAO1JY7E7EjWlnAwn8blazoOH/8Jpn1L6tqFEG4RGH2h+FpYLCRPgYRxEJVstF45neQNsPBB4+bp+gc9EqIQIvhIAveW2FFw9g+g4J9Q4ufVSEKIgCAJ3JvOvQMih8O79xp17UII4QJJ4N4UGgVfue/kE6NCCOECSeDeNvWbkJwF638GbU2+jkYIEcAkgXubyQyLfwO1xbDpL76ORggRwCSB+0L6XDjjYvjo/+DEMV9HI4QIUJLAfWXhz6GjFT74ha8jEUIEKEngvpIwDmbdDJ89B0fzfR2NECIASQL3pXk/gvB4eO8n0qxQCHHaJIH7UlgsLLgXDv0H9rzt62iEEAFGErivTV8OSWfA2vuN3g6FEMJJksB9zRwCi38Fx7+ALX/1dTRCiAAiCdwfjL8Axi+EjQ9BQ6WvoxFCBAhJ4P5i8a+MgSA2/NrXkQghAoQkcH+RNBFm3AB5T0PZLl9HI4QIAAMmcKXUU0qpcqVUYbd1Dyml9iildiilXldKxXo0ymCx4F6jw6u10qxQCDEwZ0rgzwBLeqxbB0zRWp8JfA7c6+a4glN4vDHs2v4PoGidr6MRQvi5ARO41nojUN1j3Vqtdbvj5WZAxghzlxk3Qfw4eO/H0NHm62iEEH7MHXXg3wHW9LVRKbVCKZWrlMqtqKhww+mGuBArLPolVBVB7lO+jkYI4cdcSuBKqZ8A7cBzfe2jtX5Ca52jtc5JSkpy5XTBY+JSSJ8HH/4GGqsH3l8IEZQGncCVUsuAi4Fvai133NxKKVj8a2iqgX//3tfRCCH81KASuFJqCXA3cInWutG9IQnAGLVn2rdg69+gssjX0Qgh/JAzzQhfADYBE5VSJUqpG4C/AFHAOqVUvlLqcQ/HGZy+ch+EhBn9pAghRA8hA+2gtf5GL6uf9EAsoqfIYTD3h/D+g7B/A4w7z9cRCSH8iDyJ6e9mfx9iR8Ha+3wdiRDCz0gC93cWG+TcAGWF0iJFCHEKSeCBIDHDmFd/4ds4hBB+RRJ4IIhLN+bHJYELIU6SBB4I4sYYcymBCyG6kQQeCKzhEJksJXAhxCkkgQeK+HQpgQshTiEJPFDEpUsJXAhxCknggSJ+LJwohVbpuUAIYZAEHijiO1uiHPRpGEII/yEJPFBIU0IhRA+SwANFZwlcbmQKIRwkgQeKsDgIjZESuBCiiyTwQKEUxI+RErgQoosk8EASPxaqD/g6CiGEn5AEHkji0qG2GDrafR2JEMIPSAIPJPHpYG83krgQIug5M6TaU0qpcqVUYbd1Vymldiql7EqpHM+GKLpIU0IhRDfOlMCfAZb0WFcIXAFsdHdAoh/SlFAI0Y0zY2JuVEqN6bFuN4BSykNhiV5FjQBzqJTAhRCAF+rAlVIrlFK5SqnciooKT59uaDOZjL7BpQQuhMALCVxr/YTWOkdrnZOUlOTp0w190q2sEMJBWqEEmrh0o0MrrX0diRDCxySBB5r4dGhrgPpyX0cihPAxZ5oRvgBsAiYqpUqUUjcopS5XSpUAc4B3lFLveTpQ4SBNCYUQDs60QvlGH5ted3MswhndmxKOmu3bWIQQPiVVKIEmdhQok/SJIoQIjAR+pKaJsrpmX4fhH0JCITpVqlCEEANXofiDxz7cx6rNh5kwPJJzxycxd0Iis9LjCbcGRPjuJ93KCiEIkAS+bM4Y0uLC+aioklVbDvHUx19gNZuYNjqWuRlJzM1IJHNEDGZTkDwZGpcOe972dRRCCB8LiASeMTyKjOFR3Dx/HM1tHWw9WM1HRZV8VFTJQ+/t5aH39hIbbuGccYnMzUjk3IxEUuPCfR2258SnQ2MVNNeBLdrX0QghfCQgEnh3NovZUeo2nuqsONHCx/uMZP6ffRW8U1AKQHpihJHMxycyZ1wCUTaLL8N2r+5NCVOyfRuLEMJnAi6B95QUFcplZ43ksrNGorWmqLzeSOZFFbycW8I/Nh3CbFLMGZvA988bx5yxCYHfCVdXU8IDksCFCGIBn8C7U0oxYXgUE4ZHccO56bS0d7DtUA0fFVXw6rYSrv3bFmamx3P7BRmBncjjpFtZIcQQS+A9hYaYmTMugTnjErj1/Axe/PQwj364P/ATuS0awhOlKaEQQS4g2oG7g81i5tvnpLPxrvP42Vcnc7CygWv/toWrn9jMJ/sr0YHWOZT0SihE0AuaBN5pyCTyzl4JhRBBK+gSeKeAT+Tx6VBbAu0tvo5ECOEjQZvAOwVsIo9LBzQcP+TrSIQQPhL0CbxTwCXy+LHGXG5kChG0JIH30Fci//bTW2lq7fB1eCfJCPVCBD1J4H3onsh/cuEkNhZV8MN/5mO3+0lJPCIJLBFSAhciiEkCH4DNYuameWP58dJJrCk8xu/e2+PrkAxKSVNCIYLckH6Qx51unJvOwaoG/vrvA4xJiOAbM0f5OiSIGwOVn/s6CiGEjzgzJuZTSqlypVRht3XxSql1SqkixzzOs2H6nlKKBy/JZN6EJO57o5CPiip8HZJRAj9+EOx+VDcvhPAaZ6pQngGW9Fh3D/C+1joDeN/xesgLMZt45NqzGJ8UyfdXbePzshO+DSh+LHS0Qt1R38YhhPCJARO41nojUN1j9aXAs47lZ4HL3BuW/4qyWXhq+QxsVjPLn95KxQkfPkgjI9QLEdQGexNzuNa6FMAxH9bXjkqpFUqpXKVUbkWFH1Q7uMHI2DCeXJZDVUMLN/4j13fNC6UpoRBBzeOtULTWT2itc7TWOUlJSZ4+ndecmRrLw1efxY6SGt81L4xOBVOIlMCFCFKDTeBlSqkUAMe83H0hBY4lU5K7mhf+/r293g/AHAKxo6QELkSQGmwCfwtY5lheBrzpnnACz41z07l21ige//d+Xvj0sPcDiEuXErgQQcqZZoQvAJuAiUqpEqXUDcBvgYVKqSJgoeN1UFJK8fNuzQv/U1Tp3QDixxolcH/rq0UI4XHOtEL5htY6RWtt0Vqnaq2f1FpXaa3P11pnOOY9W6kEle7NC7+3Ks+7zQvj06GlDhqD+lcgRFCSR+ndJMpm4clv5xBq8XLzQmlKKETQkgTuRqlx4V3NC2/6Ry7NbV5oXihNCYUIWpLA3Sw7zWheuN1bzQvjxhhzKYELEXQkgXvAkinJ3Lv0DFYXeKF5oSUMolKkBC5EEJLeCD3kprljOVjVyOP/3s+YhHCu8WTvhfFjofqA544vhPBLUgL3kM7eC+dmJHq+eaG0BRciKEkC9yCL2cQj35zGuKRIvv9cHsdqmz1zovgxUF8GrQ2eOb4Qwi9JAvewaJuFx6+fTmuHnZ+8XuCZwZG7mhIedP+xhRB+SxK4F6QnRnDnoom8v6ecN/KPuP8E0pRQiKAkCdxLlp+TzrRRsfzsrV2Un3BzVYo8zCNEUJIE7iVmk+L3V2bT1NbB/W8UurcqJTwebLHSEkWIICMJ3IvGD4vkhwsn8N7OMt7eUereg8sI9UIEHUngXnbjuelkp8bw07d2UlXvxv5SpCmhEEFHEriXhZhNPHRVNvXN7Tzw1k73HTg+HWqKoaPNfccUQvg1SeA+MGF4FLeeP553dpSypsBNVSlx6aA7oLbYPccTQvg9SeA+cvP8cWSOiOb+Nwupbmh1/YDSlFCIoCMJ3EcsZhMPXZlNTWMbD/7LDVUp8WONubREESJouJTAlVK3KaUKlVI7lVK3uymmoDF5RDS3nDeeN/OPsm5XmWsHi0yGEJs8jSlEEBl0AldKTQFuAmYC2cDFSqkMdwUWLG45bzxnJEfxk9cLqG104QakyWT0DS5VKEIEDVdK4JOAzVrrRq11O/Bv4HL3hBU8rCEm/nBVNlUNrfz87V2uHUyaEgoRVFxJ4IXAPKVUglIqHLgQSOu5k1JqhVIqVymVW1FR4cLphq4pI2P43vxxvLqthA17ywd/oPh0owpFRqgXIigMOoFrrXcDvwPWAe8C24H2XvZ7Qmudo7XOSUpKGnSgQ91/nT+eCcMjuffVAuqaB1mVEpcObY1G17JCiCHPpZuYWusntdbTtNbzgGqgyD1hBZ/QEDMPXZlN+Ylmfv3O7sEdRFqiCBFUXG2FMswxHwVcAbzgjqCCVXZaLDfNG8uLW4vZ+PkgqpukLbgQQcXVduCvKqV2Af8CbtFaH3dDTEHtjgsmMDYpgntfK6C+5Us1Uv2LSQNlkhuZQgQJV6tQ5mqtJ2uts7XW77srqGBms5h56MozOVrbxG9Wn2ZVSogVYlKlBC5EkJAnMf3Q9NHxfOecdJ7bcphP9p3mYMjSlFCIoCEJ3E/duWgiYxLCufu1HTScTlWK9AsuRNCQBO6nwqxmfn9lNiXHm3jovb3OvzF+LDRVQ1ONx2ITQvgHSeB+bGZ6PMvmjOGZTw7y6RfVzr1JxscUImhIAvdzdy2ZSFp8GPe8usO5cTSlKaEQQUMSuJ8Lt4bw3fnjOFDZwKGqxoHfEDfGmEsJXIghTxJ4ADgrLQ6A7SU1A+8cGgURSVICFyIISAIPABOGR2KzmMgvrnHuDXHp0i+4EEFAEngACDGbyBoZw3ZnE3h8uvSHIkQQkAQeILJTYyk8Wkdbh33gnePHQt1RaGv2fGCBwG6HyiJjLoS3Hd4MTy2FGvcPOC4JPEBkp8XS2m5n77ETA+8clw5oqDnk8bgCwsbfw19y4A/j4dUbIf8FOCFd7goPqzkMLy+HpxYbVZo1h91+ihC3H1F4xNS0WADyi2uYMjKm/527NyVMmujZwPxd7RH4z8MwZi5Ej4D9H0DBy8a25CwYdz6MvwDSZhl9yQjhqpZ6+M//waa/AArm3wPn3ArWCLefShJ4gEiNCyM+wsr24hqumz26/53lYZ6T3n8QtB0uexRiRxnVKGUFsG897PvA+CP7+GGwRhpJfvz5xtTZt7oQzrLbYceLsP5BqD8GWVfBBT8zOpjzEEngAUIpRXZqjHNNCSMSjYQU7E0JS/Jgx0tw7g+N5A3G4M8p2cY097+huQ4OfgT73of978Pna4z94tKNkvn4843EHhrpu59D+L9Dm+Dde6A0H0bmwNUrIW2mx08rCTyAZKfF8uHnFdS3tBMZ2s+vTikjAQVzSxSt4b0fG23i5/6w7/1s0XDGRcYEULX/ZDLPfx62/g1MFhg53ShJRSUbU2QyRA0/OQ+NNq67CC7HD8H6n8LO1yFqBFzxN5hypVFQ8AJJ4AEkOy0WraGgpJY54xL63zk+HcpdHOU+kO16A4o3w1f/aDzc5KyEccY0awW0t8DhzTTuXkvj/k9IqM9FnTgG7b207gkJ65bQO5P88JPz2FGQMF6SvDtobfwO2pqgtcEYB7atEVob+162t0NiBgyfYszNFtdiaDlh1HN/8hdjEJUF98LZ/+WReu7+SAIPINmpsYDxRKZTCXzvGrB3gMns+eD8SVszrHvA+GM96/rBHycklJK4GVy3y87BqrO5fvZofn7JZFRLnTFw9Iljvc/Ldho3S1vqTj1e1AijSiZjIYxdALYBbkYHs442OLYDij+F4i1Qut1Imp2JGSf6BTqFOvkecygMO8O4iT08yzHPhLDYgQ9jt8P25+H9nxu/66yvO+q5R55mPO4hCTyAxEdYGRUf7twDPXHpYG+DuiMn63+DxZbHjSZb33rTpX9eByrque7vW6hvaeeyqSNYufkQiZGh3HZBhvHHPlALn9YGR2Ivg8rPjWqZXW/BZyvBFAJpsyHjAhi/0EggwVw6b6yGkq1Gsj68BY7kQXuTsS1mFIw8C8LijRKuJQws4cZkDR942RIOaONZgGMFxk3sYwVGAeezVSdjiB11MqEnTzHmsaNP/l4OfeKo597uqOd+DtJmeP1SdedSAldK3QHciPGvrQBYrrWWp0c8KDstlryDTnQt270pYTAl8PoK2PgHmLDUKOUO0u7SOq5/8lO01ry4Yg6TUqIIMZv4v/WfkxBpHbglEBjJJn6sMY2eA9OXGSXL4k9h3zooWg/rf2ZMwVQ619q411C8+WTCrnT0eW8KgeQzYfq3YdQsSJ3pvtLt8MnGxNUn4zhxDMoKjdL+sUJHYl9NV2k9NNr4JmcJM/4BR4+EK/4OWVf6xT/cQSdwpdRI4FZgsta6SSn1T+Aa4Bk3xSZ6kZ0aw7+2H6W8rplh0ba+dzylKeF8r8TmFzb8yii5LfrFoA+RX1zDsqc+JcxiZtWNsxk/zGiB8tsrsjje0Mr9bxYSH2HlwqyU0z+42QJjzjGmC34GdaWOJo3rYNebgVs61xo6Wo2p3THvaDm53FwDJblGwi7eAo1VxvtssUYb/DO/DqNmw4hpRunZG5SC6BRjylh4cn1rA5TvNpL5sQIjwVfsgQU/dtRzeyk+J7hahRIChCml2oBw4KjrIYn+dD7Qs72kloWT+0ngMalG64lgakpYtgu2PQszVxg3qgZh84EqbnhmKwmRoTx34yzS4k/+sYaYTfzl2mlc/+QWbn8xn9gwC2ePT3Qt5ugUmHa9MfVXOh9zDpitxj0N3eGY2x3L9m7rum07ZZ8O1+LUdiO+nom5++SMhPHGt6O0mUbCTsjwWosNp1kjIDXHmPzcoBO41vqIUuoPwGGgCVirtV7bcz+l1ApgBcCoUUH0Vd5DMkfEYDYpthfXsHDy8L53NJkhbnTwNCXUGtb+xPjKO//uQR3iw73l3Lwyj1Hx4ay6cRbDe/mGE2Y18+SyGXz9r5tYsTKPF1fMHvjJWGf1Vzo/vNn4GU0mUGbj99s1NxnTKes655aT23ChFK+U8Q+kcwqxGjcDu5Y714f2vmwNh5SzIGKAm+/itLhShRIHXAqkAzXAy0qp67TWq7rvp7V+AngCICcn53RvHYsewqxmJg6Pcu6BnmAaob5ondHyY8lvITz+tN++pqCUW1/8jAnDo1h5wyziI/p+rD4m3MKz35nJ1x77hG8//SmvfPdsxiR6oPlY99K5EL1w5bvLBcAXWusKrXUb8BpwtnvCEv3JTotle3ENdvsA/w/j06H6oFFyG8o62ozSd/w4yLnhtN/+al4Jtzy/jTNTY3n+ptn9Ju9OyTE2/nHDTOwarn9qC+V1cu9eeJ8rCfwwMFspFa6UUsD5wG73hCX6MzUthrrmdg5WNfS/Y1w6tJ44ecNoqMp7xmimt+iXp90h1cpNB/nvl7czZ1wCK2+YSUyY8w94jEuK5Olvz6CqvpVlT2+lrrntNAMXwjWDTuBa6y3AK8A2jCaEJhxVJcKzsrtuZNb0v2MwDHDcdBw2/BrS58HEpaf11sf/vZ/739zJBZOG8+SyGYRbT79GMTstlr9eP5195Se48dlcmttcvFkoxGlw6fav1vqnWusztNZTtNbXa61b3BWY6FvGsCjCrWa2F9f2v2Mw9Eq48Q9GEl/8a6eb2mmt+Z+1e/ntmj1ckj2Cx66bhs0y+Ad+5mYk8T9fn8rWg9Xc+sJntDsz6IYQbuBn7XeEM8wmxZSRTvRMGDcGUEO3JUrVftjyVzjrOuOpOSdorfn527v48wf7uGZGGv939VQsZtf/DC7JHsFPL57M2l1l3PdGIXqo33cQfkEepQ9QU9NieeaTg7S227GG9JGALDZjEIOhWoWy7gGjqdpX7ndq9w675sevFfBSbjE3nJvOfRdNQrnxAZlvn5NOZX0rf9mwj8TIUO5cHOSDaQiPkwQeoLJTTw6xlpXaTzvkodqU8IuPYM/bRvKO6qc9vENbh507Xsrn7R2l3Hp+BndckOHW5N3pvxdNoKqhhb9s2EdCpJXl56S7/RxCdJIEHqCy04yknV9S038Cjx8Dn3/p+arAZrcbfX3HpMGcWwbcvbmtgx88v431u8v58YVnsGLeOI+FppTil5dlUd3QyoP/2kV8hJVLp3q/p7r2DjsV9S2U1jZTVtvM8cY24iOsDI8OZXi0jaSoULdUHYkv01pzrK6ZnUfqKDxay86jdew6Wsefrz2LaaPi3HouSeABamRsGImRxhBr1/fXsVJcOjSUG+P0DZVRZba/YHQ+9LUnjU6GBvCPTQdZv7ucX1w2pf9r5SZmk+KP15zFsqc+5c6XtxMXbmXehCS3Hb+lvYPyOiM5l9Y2cay22UjUdcb8WG0z5Sea6e8xAaUgIcLK8GibYwplWNTJ5c71CRFWTCY/74elh+a2DqobWqmqb6WyoYXq+lbqmjv/gRk/V3K0jTCr690s2+2ag1UN7Dxa55iMhF3dYHQtoBSkJ0YwbXQcVg/8w5QEHqCMIdZiB+5atrMp4fGDRheZga6l3uiLOXUGTPmaU295Z0cp2akxXknenWwWM39blsPVf93Md1fl8fxNs5maFovdrmlobaextYP6lnYaWxzz1nYaWjtoaGl3TB00trY7tnVQ19TGsTojOVc1fLnfkQirmZTYMFJibGQMSyQ5xkZyjI2UGBvJ0WHEhluobmilrK6ZsroWyuqMJN+5vKOkhsr6Lx83xKRIigolIdKK2WRCYSQlY666vTZWdn+t1Mllk0lhCzERZjUTZjFjs5gJs5oJd8xtFmP9l7Y7XoeYFTWNbVTVt1LV0HLKvLK+leqGFqocSbu+pd2p31GULYTkaOM6DYuykRwT2u0fmpHkEyOthDgSb1uHnaKyegqP1rLLkax3Ha2jodVoOmoxKyYMj+KCScPIHBHDlJHRnJEcTUR/o2e5SBJ4ADszNZYP9pZzormNKFsfD6B0Ds5bfWBoJPCP/2gMGHv1KqeaDZYcb2R7SS33LD3DC8GdKtpm4dnvzODKxzZx1eOfEGIy0XQa7cRtFhMR1hDCQ81EhVoYHh3KmamxRlJ2JOeUGCPZ9Pn772ZEbFi//ba0ddipONHSleSNBG8sV9W30KHpal2jNWi0Me++jKP/LOwnX2tNh4bytg6a2zpobO2gybHc1jH41jpmkyI+wkpChJXEyFBS48JJiDSWEyKsxrbIUBIjrUTZLFQ3tFBW18Kx2maO1TVTXmfMy+pa2FdeSfmJFjp6fG0xKUiMDCUmzMKhqkZaHU1Ew61mJqdEc+X0VDJHxDB5RDQThkf13aDAQySBB7DstBhjiLUjtZw9ro9e8YZSW/DaEvjkz0bJ28mO9NcUHAPgwimD6PrVDYZF2Xjuxlk888lBzCZFuNVMZGgI4dYQIkLNXQn6lHWhIYRbzF0lP2+xmE2MiA1jROzA1VLu0tZhp7nNkdBb7TS1Gd88OhN8k2Nda7ud2HALCd2ScrTNclrVO/ERVsYP63t4vQ67pqqhhbJa459Y9yR/vLGNrzhK1pkjohmTEIHZD6qWJIEHsK4h1or7SeBhsRAWNzSaEr7/c0AbPfU5aXVhKVNGRjMqwXd9OKfFh3P/xZN9dn5/ZjGbsJhNTn2D8DSzSTEsyqhOySIwBtSQ29ABLC7CyugEJ4ZYGwpNCUvyYMdLRqsTJ0cYOlrTxGeHa1jqo9K3EJ4mCTzAZafGOtcnSiCXwLU2mg1GDINz73D6be8WOqpPBjNyjhABQBJ4gMtOi+1qQtanhAyoOQRPXwS5TxkDyAaKo/nwxveM8RO/ch+E9l2H2dPqglImpUST7om+uoXwA1IHHuCmOh7o2V5cw6LM5N53mv1dY1SWgpfh7Ttg9Y9g3Fcg6yqYeKH/tQ9va4Kdr8PWJ+FIrjGq+MybjT5PnHSstpncQ8f574UTPBioEL4lCTzAdQ2xVtJPAg+LgwV3w/y7jAdgCl6Bwteg6CYICYOJS2DKlcbAriGh3v0Buqvab3xDyH/O6GEwcQIs/T2cebVxM/Y0vLfTqD5ZKtUnYgiTBB7gbBYzZyRHDdy1LBjtplOyjemCB43RwQtfgZ1vGCXe0BiY/FUjmafPc4yj6GEd7fD5u5D7pDEkmikEJn3VGFlnzLmDHo19dUEpE4ZHdo0oL8RQJAl8CMhOi+Vf249it2vn28WaTDB6jjEt+R188SEUvAo734TPVhk3DDMvh6wrjace3d3x04ljsO0fxmg6dUcgeiScd58x/mNUH98knFR+oplPD1Zz2/mDG5leiEAhCXwImJoay/NbDvNFVQPjkgZR4jSHwPgLjOni/4WitUY1S94z8OlfIXa0MdpNVAqEJxhTRKJjOd4ouZucuB+uNRz8yKjb3vM22NuNuvgLH4KMxUYcbvDezjK0ltYnYuhzZVT6icBL3VaNBR7QWj/salDi9HQNsVZcM7gE3p0lDCZfakzNdbDnHaOaJe9ZaG/q/T3KfDKxhydARMKpr8MToaEC8p42xq4Mi4PZ34PpyyHB/T0DrikoZVxSBBlSfSKGuEEncK31XmAqgFLKDBwBXndPWOJ0jB8W6RhirYYrpqW678C2aJj6DWMCaG0wBkjunBq6LTdWOubVULEXGiqhqdroGKNT6gy47HHIvMypXgQHo7K+hc0HqrjlvPEe6e9bCH/iriqU84H9WutDbjqeOA1mkyJrZAz5JU7cyHSFNcKYnHwSErsdmmuMxK5MHilt97R2Zxl2qT4RQcJdD/JcA7zQ2wal1AqlVK5SKreiosJNpxM9TU2LZffROlra/WhUdJPJqCNPzPBK8gZYU1hKemIEZyQ7/8CPEIHK5QSulLIClwAv97Zda/2E1jpHa52TlOS+Tu3FqbLTYmntsLOn9ISvQ/GZ4w2tfLK/iqVTkqX6RAQFd5TAlwLbtNZlbjiWGKSuG5kD9YsyhK3ddYwOu5bqExE03JHAv0Ef1SfCe0bE2EiMDCV/oJ4Jh7DVBccYFR9O5ohoX4cihFe4lMCVUuHAQuA194QjBkspxdS0mIG7lh2iahvb+HhfJUuzpPpEBA+XErjWulFrnaC19nDzB+GM7NRY9lc0UNfc5utQvG7d7jLa7dpnI+8I4QvyJOYQ0lkPXlBSyznj+xihZ4haXVDKyNgwzkwNjJFUhoK2tjZKSkpobu6nK2NxWmw2G6mpqVgszo1QJAl8COlMXvnFNUGVwOua2/ioqIJvnz1Gqk+8qKSkhKioKMaMkevuDlprqqqqKCkpIT093an3yIAOQ0hsuJX0xAh2BFlLlPd3l9HWoaXrWC9rbm4mISFBkrebKKVISEg4rW80ksCHmOzUGOe6lh1C3tlxjJQYG1MdgzwL75Hk7V6nez0lgQ8x2WmxHKtr5lhtcNRLnmhuY2NRBUumJDvfla4QQ4Qk8CEm2B7o+WBPOa3tdi6S6hMRhCSBDzGTU6IJMamgaQ++puAYw6JCmTYqztehCB+oqanh0UcfPe33XXjhhdTU1Lg/IC+TVihDjM1iZlJKdFCUwBta2tmwt5xrZqRJ9YmPPfivnew6WufWY04eEc1Pv5rZ7z6dCfz73//+Kes7Ojowm/seEnD16tVuidHXpAQ+BGWnxbCjuBa7Xfs6FI/asLeclna7tD4JYvfccw/79+9n6tSpzJgxg/POO49rr72WrKwsAC677DKmT59OZmYmTzzxRNf7xowZQ2VlJQcPHmTSpEncdNNNZGZmsmjRIpqa+hi4BFiwYAF33HEH8+bNY9KkSWzdupUrrriCjIwM7rvvPgAaGhq46KKLyM7OZsqUKbz0kjHuTV5eHvPnz2f69OksXryY0tJS1y+A1tpr0/Tp07XwvH9uPaxH3/22Lio74etQPOr7q/L09F+s0+0ddl+HEpR27drl6xD0F198oTMzM7XWWm/YsEGHh4frAwcOdG2vqqrSWmvd2NioMzMzdWVlpdZa69GjR+uKigr9xRdfaLPZrD/77DOttdZXXXWVXrlyZZ/nmz9/vr7rrru01lo//PDDOiUlRR89elQ3NzfrkSNH6srKSv3KK6/oG2+8ses9NTU1urW1Vc+ZM0eXl5drrbV+8cUX9fLly3s9R2/XFcjVveRUqUIZgqZ2G2JtqI7K3tTawQd7yvna9JGYpfpEOMycOfOUh2D+9Kc/8frrxkBhxcXFFBUVkZCQcMp70tPTmTp1KgDTp0/n4MGD/Z7jkksuASArK4vMzExSUoxvgGPHjqW4uJisrCzuvPNO7r77bi6++GLmzp1LYWEhhYWFLFy4EDCqeDrf5wpJ4EPQ2KRIIkND2F5Sw9emu3GINT/y4d5ymto6pO8TcYqIiIiu5Q8//JD169ezadMmwsPDWbBgQa8PyYSGhnYtm83mfqtQuu9vMplOea/JZKK9vZ0JEyaQl5fH6tWruffee1m0aBGXX345mZmZbNq0ydUf8RRSBz4EdQ6xNpRboqwuPEZ8hJWZ6fG+DkX4UFRUFCdO9D6ISW1tLXFxcYSHh7Nnzx42b97slZiOHj1KeHg41113HXfeeSfbtm1j4sSJVFRUdCXwtrY2du7c6fK5pAQ+RGWnxfLkfw7Q0t5BaEjfd+MDUXNbBx/sLuOSqSMJMUsZJJglJCRwzjnnMGXKFMLCwhg+fHjXtiVLlvD4449z5plnMnHiRGbPnu2VmAoKCvjRj36EyWTCYrHw2GOPYbVaeeWVV7j11lupra2lvb2d22+/nczM/lvZDEQZ9ePekZOTo3Nzc712vmD2bmEp3121jTduOaerTnyoeG/nMW5emcfKG2YyN0OG6fOV3bt3M2nSJF+HMeT0dl2VUnla65ye+0rxZYjK7nYjc6hZU1BKbLiF2WMTBt5ZiCFMqlCGqORoG8OiQodcAm9p72D97nIuzErGItUnwkNuueUWPv7441PW3XbbbSxfvtxHEfVOEvgQpZQiOy2W/CH2ROZ/iiqpb2mXgYuFRz3yyCO+DsEpro6JGauUekUptUcptVspNcddgQnXTU2L5UBFA7VNQ2eItXcKSom2hXD2uOAZsEKIvrj6HfSPwLta6zOAbGC36yEJd8l29I9dUDI0+gdvbbezblcZCycnYw2R6hMhBv1XoJSKBuYBTwJorVu11jVuiku4QZZjiLWh0rHVx/srOdHczoVZyb4ORQi/4EoxZixQATytlPpMKfV3pVREz52UUiuUUrlKqdyKigoXTidOV0yYhbFJEeQPkRuZq3eUEhUawrkZUn0iDIPtThbg4YcfprGx0c0ReZcrCTwEmAY8prU+C2gA7um5k9b6Ca11jtY6JylJ2ux629TUWPKLa/Bme39PaOuws3ZXGRdMHj7kHkwSgxfsCdyVViglQInWeovj9Sv0ksCFb2WnxfLaZ0c4VtdMSkyYr8MZtE37q6htamPpFKk+8Utr7oFjBe49ZnIWLP1tv7t070524cKFDBs2jH/+85+0tLRw+eWX8+CDD9LQ0MDXv/51SkpK6Ojo4P7776esrIyjR49y3nnnkZiYyIYNG3o9fmRkJLfccgvr168nLi6OX//619x1110cPnyYhx9+mEsuuYSdO3eyfPlyWltbsdvtvPrqq2RkZLBq1Sr+9Kc/0drayqxZs3j00Uf77aN8MAZdAtdaHwOKlVITHavOB3a5JSrhNkPlgZ41haVEWM3MmyDf4sRJv/3tbxk3bhz5+fksXLiQoqIiPv30U/Lz88nLy2Pjxo28++67jBgxgu3bt1NYWMiSJUu49dZbGTFiBBs2bOgzeYPRt/eCBQvIy8sjKiqK++67j3Xr1vH666/zwAMPAPD4449z2223kZ+fT25uLqmpqezevZuXXnqJjz/+mPz8fMxmM88995zbf35X24H/F/CcUsoKHAD8q5W7YFJKFBazIr+4liUB2nNfe4ed93aWcf6k4dgsUn3ilwYoKXvD2rVrWbt2LWeddRYA9fX1FBUVMXfu3C917+osq9XKkiVLAKP72NDQUCwWC1lZWV3dzs6ZM4df/epXlJSUdA3u8P7775OXl8eMGTMAaGpqYtiwYe79gXExgWut84EvPZ8v/EdoiJnJKdF+VwLvsGvqW9ppcEz1jslY7uha19DSztGaJqobWqX1ieiX1pp7772Xm2+++Uvbenbv2ll6HojFYkEpo7/57t3HdnYdC3Dttdcya9Ys3nnnHRYvXszf//53tNYsW7aM3/zmN2766XonT2IGgey0WF749DDfeGIzafFhpMWFkxYf3rWcGBnqljEltdZU1rdScryRIzVNHDneRMnxJo7UNFFW19wtQbfT3GZ36pghJkVEaAjZqTHMn+D+EowIbN27k128eDH3338/3/zmN4mMjOTIkSNYLBba29uJj4/nuuuuIzIykmeeeeaU9yYmutaq6cCBA4wdO5Zbb72VAwcOsGPHDhYtWsSll17KHXfcwbBhw6iurubEiROMHj3a1R/5FJLAg8D1s0dT19TG4epGNuytoOJEyynbQ0NMjIzrTOzdEnxcOKlxYcSGG6WQDrumrK6ZIzVNRpJ2JOeS401dyy3tpybmaFsII+PCSY4OZVxSJBGhIUSGmh1zY4roNo8INZ+yLjTE1FUCEqKn7t3JLl26lGuvvZY5c4wHwiMjI1m1ahX79u37UveuACtWrGDp0qWkpKT0Ww8+kJdeeolVq1ZhsVhITk7mgQceID4+nl/+8pcsWrQIu92OxWLhkUcecXsCl+5kg1BzWwclxxspPt5ESbUxL65upPh4I8XVTV969D4yNISYMAtldc209xgoOTHSysjYMEbGhZEaF24sO16PjAsj2mbx5o8mvEi6k/WM0+lOVkrgQchmMTN+WBTjh0X1ur2uuc1I6NVGSbvkuJHUU2JsRpKOC+tK1GFWuakohK9IAhdfEm2zkDkihswRMb4ORQivmDVrFi0tp1Ytrly5kqysLB9F5BxJ4EKIoLdly5aBd/JD0qWbEGLQAr2LBn9zutdTErgQYlBsNhtVVVWSxN1Ea01VVRU2m83p90gVihBiUFJTUykpKUF6GXUfm81Gamqq0/tLAhdCDIrFYiE9Pd3XYQQ1qUIRQogAJQlcCCEClCRwIYQIUF59lF4pVQEcGuTbE4FKN4bjbhKfayQ+10h8rvPnGEdrrb/UGb5XE7grlFK5vfUF4C8kPtdIfK6R+FwXCDH2JFUoQggRoCSBCyFEgAqkBP6ErwMYgMTnGonPNRKf6wIhxlMETB24EEKIUwVSCVwIIUQ3ksCFECJA+V0CV0otUUrtVUrtU0rd08t2pZT6k2P7DqXUNC/GlqaU2qCU2q2U2qmUuq2XfRYopWqVUvmOybnhr90X40GlVIHj3F8av87H129it+uSr5SqU0rd3mMfr14/pdRTSqlypVRht3XxSql1Sqkixzyuj/f2+1n1YHwPKaX2OH5/ryulYvt4b7+fBQ/G9zOl1JFuv8ML+3ivr67fS91iO6iUyu/jvR6/fi7TWvvNBJiB/cBYwApsByb32OdCYA2ggNnAFi/GlwJMcyxHAZ/3Et8C4G0fXsODQGI/2312/Xr5XR/DeEDBZ9cPmAdMAwq7rfs9cI9j+R7gd33E3+9n1YPxLQJCHMu/6y0+Zz4LHozvZ8CdTvz+fXL9emz/H+ABX10/Vyd/K4HPBPZprQ9orVuBF4FLe+xzKfAPbdgMxCqlUrwRnNa6VGu9zbF8AtgNjPTGud3IZ9evh/OB/VrrwT6Z6xZa641AdY/VlwLPOpafBS7r5a3OfFY9Ep/Weq3Wut3xcjPgfP+jbtbH9XOGz65fJ6WUAr4OvODu83qLvyXwkUBxt9clfDlBOrOPxymlxgBnAb2NxTRHKbVdKbVGKZXp3cjQwFqlVJ5SakUv2/3i+gHX0Pcfji+vH8BwrXUpGP+0gWG97OMv1/E7GN+oejPQZ8GTfuCo4nmqjyoof7h+c4EyrXVRH9t9ef2c4m8JXPWyrmc7R2f28SilVCTwKnC71rqux+ZtGNUC2cCfgTe8GRtwjtZ6GrAUuEUpNa/Hdn+4flbgEuDlXjb7+vo5yx+u40+AduC5PnYZ6LPgKY8B44CpQClGNUVPPr9+wDfov/Ttq+vnNH9L4CVAWrfXqcDRQezjMUopC0byfk5r/VrP7VrrOq11vWN5NWBRSiV6Kz6t9VHHvBx4HeOranc+vX4OS4FtWuuynht8ff0cyjqrlRzz8l728fXncBlwMfBN7aiw7cmJz4JHaK3LtNYdWms78Lc+zuvr6xcCXAG81Nc+vrp+p8PfEvhWIEMple4opV0DvNVjn7eAbzlaU8wGaju/7nqao87sSWC31vp/+9gn2bEfSqmZGNe4ykvxRSilojqXMW52FfbYzWfXr5s+Sz6+vH7dvAUscywvA97sZR9nPqseoZRaAtwNXKK1buxjH2c+C56Kr/s9lcv7OK/Prp/DBcAerXVJbxt9ef1Oi6/vovacMFpJfI5xh/onjnXfBb7rWFbAI47tBUCOF2M7F+Nr3g4g3zFd2CO+HwA7Me6qbwbO9mJ8Yx3n3e6Iwa+un+P84RgJOabbOp9dP4x/JKVAG0ap8AYgAXgfKHLM4x37jgBW9/dZ9VJ8+zDqjzs/g4/3jK+vz4KX4lvp+GztwEjKKf50/Rzrn+n8zHXb1+vXz9VJHqUXQogA5W9VKEIIIZwkCVwIIQKUJHAhhAhQksCFECJASQIXQogAJQlcCCEClCRwIYQIUP8PVw4nG3LBmQUAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 创建随机数种子\n",
    "torch.manual_seed(29)  \n",
    "\n",
    "# 实例化模型\n",
    "relu_model3 = ReLU_class3(bias=False)              \n",
    "\n",
    "# 核心参数\n",
    "num_epochs = 20\n",
    "lr = 0.03\n",
    "\n",
    "# 模型训练\n",
    "train_l, test_l = model_train_test(relu_model3, \n",
    "                                   train_loader,\n",
    "                                   test_loader,\n",
    "                                   num_epochs = num_epochs, \n",
    "                                   criterion = nn.MSELoss(), \n",
    "                                   optimizer = optim.SGD, \n",
    "                                   lr = 0.03, \n",
    "                                   cla = False, \n",
    "                                   eva = mse_cal)\n",
    "\n",
    "# 绘制图像，查看MSE变化情况\n",
    "plt.plot(list(range(num_epochs)), train_l, label='train_mse')\n",
    "plt.plot(list(range(num_epochs)), test_l, label='test_mse')\n",
    "plt.legend(loc = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们发现，初始参数值的选取不仅会影响模型收敛速度，甚至在某些情况下还会影响模型的最终表现。造成此现象的根本原因还是在于神经网络模型在进行训练时，不确定性过多，而在一个拥有诸多不确定性的系统中再加上不确定的初始参数，初始参数的不确定性会被这个系统放大。并且，值得一提的是，每一个epoch中的每一次迭代并不是在一个损失函数上一步步下降的，当我们使用小批量梯度下降算法时，带入不同批的数据，实际创建的损失函数也会不同。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch24",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
